Repository: ranaroussi/yfinance

Issue:
Scraper error "TypeError: string indices must be integers"  - Yahoo decrypt fail
## Updates
### 2023 January 13
By the time of posting the issue (2023 January 12), the issue only occured sometimes. The library is now (2023 January 13) completely broken and I am unable to retrieve any stock informatio
### 2023 January 14
Fix has been merged to the branch `dev`

## Info about your system:
yfinance version: 0.2.3
Operating system: macOS Monteray 12.0.1
### Snippet that can recreate the error
```
stock = yf.Ticker("^GSPC")
info = stock.info
```
## Error
Message:`TypeError: string indices must be integers`
It seems to be a problem where the scraper is not scraping the correct information, leading to a crash.
### Traceback:
```
Traceback (most recent call last):
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/apscheduler/executors/base_py3.py", line 30, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
  File "/home/2022/szhang139/repos/STONK/src/main.py", line 61, in notify
    market = get_major_index(f'Market Close - {daytime.today_date()}')
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 63, in get_major_index
    sp500 = get_stock('^GSPC')
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 41, in get_stock
    stock_info = get_stock_info(stock_name)
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 8, in get_stock_info
    info = stock.info
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/ticker.py", line 138, in info
    return self.get_info()
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/base.py", line 894, in get_info
    data = self._quote.info
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 27, in info
    self._scrape(self.proxy)
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 58, in _scrape
    quote_summary_store = json_data['QuoteSummaryStore']
```
### Frequency
The error occurs in no apparent pattern. Every time it occurs, it seem to persist for some range of time before it recovers back to normal.
n. 



Discussion:
I also have the exact same problem as you, good to know that im not alone. The yfinance ticker module seems very unstable at the moment.
Same issue here.

Traceback (most recent call last):
  File "/home/gulder/PycharmProjects/YPUpdater/YPFunctions.py", line 171, in updatepricedata
    beta = yf.Ticker(ticker).info.get('beta')
  File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/ticker.py", line 138, in info
    return self.get_info()
  File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/base.py", line 894, in get_info
    data = self._quote.info
  File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/scrapers/quote.py", line 27, in info
    self._scrape(self.proxy)
  File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/scrapers/quote.py", line 58, in _scrape
    quote_summary_store = json_data['QuoteSummaryStore']
TypeError: string indices must be integers
Are you accessing `info` in a loop of many tickers? Spamming `info` is a great way to get temporarily blocked, which I suspect.

If yes, what key are you accessing? Might be better way to get data.
> Are you accessing `info` in a loop of many tickers? Spamming `info` is a great way to get temporarily blocked, which I suspect.
> 
> If yes, what key are you accessing? Might be better way to get data.

I have the same error, and it does not matter if I looping through 4 or 503 stocks.. 
```
data = yf.download(tickers, period = "max", interval = "1mo", progress=True)
stocks = []
for ticker in tickers:
    df = data.xs(ticker, level=1, axis=1).copy()
    df.dropna(inplace=True)
    if df.empty:
        continue
    info = yf.Ticker(ticker).info
    div = info.get("dividendYield")  
    mcap = info.get("marketCap") 
    PE = info.get("trailingPE")
    if div is not None and div >= 0.04 and mcap is not None and mcap > 1E10 and PE is not None and PE < 15:
        stocks.append(ticker)
```
```
Traceback (most recent call last):

  File "C:\Users\untitled0.py", line 18, in <module>
    info = yf.Ticker(ticker).info

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\ticker.py", line 138, in info
    return self.get_info()

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\base.py", line 894, in get_info
    data = self._quote.info

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\scrapers\quote.py", line 27, in info
    self._scrape(self.proxy)

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\scrapers\quote.py", line 58, in _scrape
    quote_summary_store = json_data['QuoteSummaryStore']

TypeError: string indices must be integers
```
Yes me too. It seems it went down hill over the last few moments. I can now only run about 1 at a time.
I get the error on financials with some symbols and on balancesheet data with other symbols, but none on price data or dividends
> 

Yes, I am, loop of around 450 tickers.
I am accessing:
yf.Ticker(ticker).info.get('beta') and
yf.Ticker(ticker).info.get('dividendRate')

Indeed, I also just encountered the same thing. 
In the specific tool, I'm suing it for a single ticker, and  I get this issue as well

> # Retrieve fundamental and technical data from yfinance
        ticker_data = yf.Ticker(ticker)
        time.sleep(2)  # just added today, doesn't help
        ticker_info = ticker_data.info

I tried adding sleep (above), but it didn't help either.

The weird part, is that for a short time when I wrote the code line by line in the Python Console (pycharm) it worked, but now even in the console I get ticker_info as 'None'

Same failure, im def not ip blocked and didn't spam. It started yesterday and today I get the same failure the whole time
Just to +1 that I am having the same issue. Just checking 1 ticker, once a minute. 
I believe the cause of this relates to the underlying data structure this library scrapes from [Yahoo Finance](https://finance.yahoo.com/) has changed. They introduced encryption on the ticker data before the turn of the year, which was fixed here: https://github.com/ranaroussi/yfinance/pull/1253/commits/8e5f0984af347afda6be74b27a989422e49a975b. This time around they changed the parameters on the encryption.
In case it helps, these were the decryption params Yahoo was providing:
```
_cs = '3b1885bff81b'
_cr = {"words":[1975313454,-1113096907,-1675002031,-87263202], "sigBytes":16}
```
# We know the decryption is failing. Please keep future comments related to debugging or fixing.
Where Yahoo used to provide the decryption parameters, there is this new key-value. It appears to be constant.
```
'93634354fa08860f003e': '4f566bf243f93d68718581c573aee0ff86456eef74bf9c33244c765b6ae2470059db233bf70a4fb3c16c24047befc4e993e717df7cbd4f1bebddb9649abdf5e3'
```
Any ideas?
I think `93634354fa08860f003e` key/value will change with time..

             encrypted_stores = data['context']['dispatcher']['stores']
    -        _cs = data["_cs"]
    -        _cr = data["_cr"]
    -
    -        _cr = b"".join(int.to_bytes(i, length=4, byteorder="big", signed=True) for i in json.loads(_cr)["words"])
    -        password = hashlib.pbkdf2_hmac("sha1", _cs.encode("utf8"), _cr, 1, dklen=32).hex()
    +        password_key = next(key for key in data.keys() if key not in ["context", "plugins"])
    +        password = data[password_key]
 
             encrypted_stores = b64decode(encrypted_stores)

and it seems this

    -    plaintext = unpad(plaintext, 16, style="pkcs7")
    +    plaintext = unpad(plaintext, 128, style="pkcs7")
That is working for me though I had to leave the unpad block_size arg at 16 when using usePycryptodome. But also works without usePycryptodome.
Can you submit a pull request? If a Github newbie see #1084 

Resolving patch:
diff --git a/yfinance/data.py b/yfinance/data.py
--- a/yfinance/data.py
+++ b/yfinance/data.py
@@ -46,13 +46,33 @@ def wrapped(*args, **kwargs):
     return wrapped
 
 
-def decrypt_cryptojs_aes(data):
+def decrypt_cryptojs_aes_stores(data):
     encrypted_stores = data['context']['dispatcher']['stores']
-    _cs = data["_cs"]
-    _cr = data["_cr"]
 
-    _cr = b"".join(int.to_bytes(i, length=4, byteorder="big", signed=True) for i in json.loads(_cr)["words"])
-    password = hashlib.pbkdf2_hmac("sha1", _cs.encode("utf8"), _cr, 1, dklen=32).hex()
+    if "_cs" in data and "_cr" in data:
+        _cs = data["_cs"]
+        _cr = data["_cr"]
+        _cr = b"".join(int.to_bytes(i, length=4, byteorder="big", signed=True) for i in json.loads(_cr)["words"])
+        password = hashlib.pbkdf2_hmac("sha1", _cs.encode("utf8"), _cr, 1, dklen=32).hex()
+    else:
+        # Currently assume one extra key in dict, which is password. Print error if 
+        # more extra keys detected.
+        new_keys = [k for k in data.keys() if k not in ["context", "plugins"]]
+        l = len(new_keys)
+        if l == 0:
+            return None
+        elif l == 1 and isinstance(data[new_keys[0]], str):
+            password_key = new_keys[0]
+        else:
+            msg = "Yahoo has again changed data format, yfinance now unsure which key(s) is for decryption:"
+            k = new_keys[0]
+            k_str = k if len(k) < 32 else k[:32-3]+"..."
+            msg += f" '{k_str}'->{type(data[k])}"
+            for i in range(1, len(new_keys)):
+                msg += f" , '{k_str}'->{type(data[k])}"
+            raise Exception(msg)
+        password_key = new_keys[0]
+        password = data[password_key]
 
     encrypted_stores = b64decode(encrypted_stores)
     assert encrypted_stores[0:8] == b"Salted__"
@@ -98,7 +118,10 @@ def EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="m
         key, iv = key_iv[:keySize], key_iv[keySize:final_length]
         return key, iv
 
-    key, iv = EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="md5")
+    try:
+        key, iv = EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="md5")
+    except:
+        raise Exception("yfinance failed to decrypt Yahoo data response")
 
     if usePycryptodome:
         cipher = AES.new(key, AES.MODE_CBC, iv=iv)
@@ -176,15 +199,16 @@ def get_json_data_stores(self, sub_page: str = None, proxy=None) -> dict:
 
         data = json.loads(json_str)
 
-        if "_cs" in data and "_cr" in data:
-            data = decrypt_cryptojs_aes(data)
-
-        if "context" in data and "dispatcher" in data["context"]:
-            # Keep old code, just in case
-            data = data['context']['dispatcher']['stores']
+        stores = decrypt_cryptojs_aes_stores(data)
+        if stores is None:
+            # Maybe Yahoo returned old format, not encrypted
+            if "context" in data and "dispatcher" in data["context"]:
+                stores = data['context']['dispatcher']['stores']
+        if stores is None:
+            raise Exception(f"{self.ticker}: Failed to extract data stores from web request")
 
         # return data
-        new_data = json.dumps(data).replace('{}', 'null')
+        new_data = json.dumps(stores).replace('{}', 'null')
         new_data = re.sub(
             r'{[\'|\"]raw[\'|\"]:(.*?),(.*?)}', r'\1', new_data)
 


Test patch: