Repository: ranaroussi/yfinance (python)

Issue:
fix_Yahoo_returning_live_separate() fails for "1wk" interval if a new year occurs midweek
### Describe bug

This week, I noticed the final row in my weekly dataframe is extraneous and does not represent a full week. After some investigation, I saw that "fix_Yahoo_returning_live_separate()", called in the "history" function, was created to address this issue. 

The comment in the "fix_Yahoo_returning_live_separate()" function claims that yfinance sometimes returns live data as a separate row when the market is still open (or for the most recent interval), which can result in inconsistent intervals in the downloaded data. This appears to be a known quirk with yfinance, especially when working with weekly or monthly intervals. I found that "fix_Yahoo_returning_live_separate()" does this for the weekly interval by checking if the year and week are the same (on line 620 and 621):   
```
 if interval == "1wk":
       ast_rows_same_interval = dt1.year == dt2.year and dt1.week == dt2.week
```
However, a new year may fall midweek, such as the start of 2025. So, unfortunately, I am seeing this extra row when I currently get weekly data (since today is 1/4/2025). 

Suggested Solution:
To correctly handle this scenario, the function should use ISO week numbers (isocalendar().week), which account for weeks spanning across years.
```
if interval == "1wk":
    # Check if both rows fall in the same ISO calendar week -- might have to use "isocalendar().week" instead.
    last_rows_same_interval = (dt1.isocalendar()[1] == dt2.isocalendar()[1])
```
This approach ensures rows are merged based on their true calendar week, not limited by year boundaries.

It would be great if this fix can be incorporated into both Ticker.history() and yf.download(). I would love a fast resolution as this is important to my personal project. Thank you!

### Simple code that reproduces your problem

Use Ticker.history() or yf.download() with a "1wk" interval (might have to be before next week, 1/6/2025) to see that there is an extraneous row for 2025-01-03 when it should be merged into the week of 2024-12-30.

For example, I see the data below:
```
2024-12-30   110.690002  115.480003  109.570000  115.199997 ...
2025-01-03   112.480003  115.459999  112.290001  115.199997 ...
```
These two rows are part of the same ISO calendar week, yet they remain separate because the function assumes week boundaries are confined to the same year.

### Debug log

I don't see any new output when I enable_debug_mode().

### Bad data proof

_No response_

### `yfinance` version

0.2.51

### Python version

_No response_

### Operating system

_No response_

Pull request:
Fix fix_Yahoo_returning_live_separate()
Fixes #2206 
Also fix post-market interval being discarded.

Bonus: adding the dropped row to metadata as "lastTrade", looks like just time and single price is useful.

Patch:
diff --git a/yfinance/scrapers/history.py b/yfinance/scrapers/history.py
index d84e0cd..32d9c75 100644
--- a/yfinance/scrapers/history.py
+++ b/yfinance/scrapers/history.py
@@ -384,7 +384,9 @@ class PriceHistory:
             msg = f'{self.ticker}: OHLC after combining events: {df.index[0]} -> {df.index[-1]}'
         logger.debug(msg)
 
-        df = utils.fix_Yahoo_returning_live_separate(df, params["interval"], tz_exchange, repair=repair, currency=currency)
+        df, last_trade = utils.fix_Yahoo_returning_live_separate(df, params["interval"], tz_exchange, repair=repair, currency=currency)
+        if last_trade is not None:
+            self._history_metadata['lastTrade'] = {'Price':last_trade['Close'], "Time":last_trade.name}
 
         df = df[~df.index.duplicated(keep='first')]  # must do before repair
 
diff --git a/yfinance/utils.py b/yfinance/utils.py
index 201e987..deb67a3 100644
--- a/yfinance/utils.py
+++ b/yfinance/utils.py
@@ -599,15 +599,40 @@ def fix_Yahoo_returning_prepost_unrequested(quotes, interval, tradingPeriods):
     return quotes
 
 
-def fix_Yahoo_returning_live_separate(quotes, interval, tz_exchange, repair=False, currency=None):
+def _dts_in_same_interval(dt1, dt2, interval):
+    # Check if second date dt2 in interval starting at dt1
+
+    if interval == '1d':
+        last_rows_same_interval = dt1.date() == dt2.date()
+    elif interval == "1wk":
+        last_rows_same_interval = (dt2 - dt1).days < 7
+    elif interval == "1mo":
+        last_rows_same_interval = dt1.month == dt2.month
+    elif interval == "3mo":
+        shift = (dt1.month % 3) - 1
+        q1 = (dt1.month - shift - 1) // 3 + 1
+        q2 = (dt2.month - shift - 1) // 3 + 1
+        year_diff = dt2.year - dt1.year
+        quarter_diff = q2 - q1 + 4*year_diff
+        last_rows_same_interval = quarter_diff == 0
+    else:
+        last_rows_same_interval = (dt2 - dt1) < _pd.Timedelta(interval)
+    return last_rows_same_interval
+
+
+def fix_Yahoo_returning_live_separate(quotes, interval, tz_exchange, prepost, repair=False, currency=None):
     # Yahoo bug fix. If market is open today then Yahoo normally returns
     # todays data as a separate row from rest-of week/month interval in above row.
     # Seems to depend on what exchange e.g. crypto OK.
     # Fix = merge them together
-    n = quotes.shape[0]
-    if n > 1:
-        dt1 = quotes.index[n - 1]
-        dt2 = quotes.index[n - 2]
+
+    if interval[-1] not in ['m', 'h']:
+        prepost = False
+
+    dropped_row = None
+    if len(quotes) > 1:
+        dt1 = quotes.index[-1]
+        dt2 = quotes.index[-2]
         if quotes.index.tz is None:
             dt1 = dt1.tz_localize("UTC")
             dt2 = dt2.tz_localize("UTC")
@@ -618,25 +643,23 @@ def fix_Yahoo_returning_live_separate(quotes, interval, tz_exchange, repair=Fals
             # - exception is volume, *slightly* greater on final row (and matches website)
             if dt1.date() == dt2.date():
                 # Last two rows are on same day. Drop second-to-last row
+                dropped_row = quotes.iloc[-2]
                 quotes = _pd.concat([quotes.iloc[:-2], quotes.iloc[-1:]])
         else:
-            if interval == "1wk":
-                last_rows_same_interval = dt1.year == dt2.year and dt1.week == dt2.week
-            elif interval == "1mo":
-                last_rows_same_interval = dt1.month == dt2.month
-            elif interval == "3mo":
-                last_rows_same_interval = dt1.year == dt2.year and dt1.quarter == dt2.quarter
-            else:
-                last_rows_same_interval = (dt1 - dt2) < _pd.Timedelta(interval)
-
-            if last_rows_same_interval:
+            if _dts_in_same_interval(dt2, dt1, interval):
                 # Last two rows are within same interval
-                idx1 = quotes.index[n - 1]
-                idx2 = quotes.index[n - 2]
+                idx1 = quotes.index[-1]
+                idx2 = quotes.index[-2]
                 if idx1 == idx2:
                     # Yahoo returning last interval duplicated, which means
                     # Yahoo is not returning live data (phew!)
-                    return quotes
+                    return quotes, None
+
+                if prepost:
+                    # Possibly dt1 is just start of post-market
+                    if dt1.second == 0:
+                        # assume post-market interval
+                        return quotes, None
 
                 ss = quotes['Stock Splits'].iloc[-2:].replace(0,1).prod()
                 if repair:
@@ -659,31 +682,30 @@ def fix_Yahoo_returning_live_separate(quotes, interval, tz_exchange, repair=Fals
                             for c in const._PRICE_COLNAMES_:
                                 quotes.loc[idx2, c] *= 0.01
 
-                # quotes.loc[idx2, 'Stock Splits'] = 2  # wtf? why doing this?
-
                 if _np.isnan(quotes.loc[idx2, "Open"]):
-                    quotes.loc[idx2, "Open"] = quotes["Open"].iloc[n - 1]
+                    quotes.loc[idx2, "Open"] = quotes["Open"].iloc[-1]
                 # Note: nanmax() & nanmin() ignores NaNs, but still need to check not all are NaN to avoid warnings
-                if not _np.isnan(quotes["High"].iloc[n - 1]):
-                    quotes.loc[idx2, "High"] = _np.nanmax([quotes["High"].iloc[n - 1], quotes["High"].iloc[n - 2]])
+                if not _np.isnan(quotes["High"].iloc[-1]):
+                    quotes.loc[idx2, "High"] = _np.nanmax([quotes["High"].iloc[-1], quotes["High"].iloc[-2]])
                     if "Adj High" in quotes.columns:
-                        quotes.loc[idx2, "Adj High"] = _np.nanmax([quotes["Adj High"].iloc[n - 1], quotes["Adj High"].iloc[n - 2]])
+                        quotes.loc[idx2, "Adj High"] = _np.nanmax([quotes["Adj High"].iloc[-1], quotes["Adj High"].iloc[-2]])
 
-                if not _np.isnan(quotes["Low"].iloc[n - 1]):
-                    quotes.loc[idx2, "Low"] = _np.nanmin([quotes["Low"].iloc[n - 1], quotes["Low"].iloc[n - 2]])
+                if not _np.isnan(quotes["Low"].iloc[-1]):
+                    quotes.loc[idx2, "Low"] = _np.nanmin([quotes["Low"].iloc[-1], quotes["Low"].iloc[-2]])
                     if "Adj Low" in quotes.columns:
-                        quotes.loc[idx2, "Adj Low"] = _np.nanmin([quotes["Adj Low"].iloc[n - 1], quotes["Adj Low"].iloc[n - 2]])
+                        quotes.loc[idx2, "Adj Low"] = _np.nanmin([quotes["Adj Low"].iloc[-1], quotes["Adj Low"].iloc[-2]])
 
-                quotes.loc[idx2, "Close"] = quotes["Close"].iloc[n - 1]
+                quotes.loc[idx2, "Close"] = quotes["Close"].iloc[-1]
                 if "Adj Close" in quotes.columns:
-                    quotes.loc[idx2, "Adj Close"] = quotes["Adj Close"].iloc[n - 1]
-                quotes.loc[idx2, "Volume"] += quotes["Volume"].iloc[n - 1]
-                quotes.loc[idx2, "Dividends"] += quotes["Dividends"].iloc[n - 1]
+                    quotes.loc[idx2, "Adj Close"] = quotes["Adj Close"].iloc[-1]
+                quotes.loc[idx2, "Volume"] += quotes["Volume"].iloc[-1]
+                quotes.loc[idx2, "Dividends"] += quotes["Dividends"].iloc[-1]
                 if ss != 1.0:
                     quotes.loc[idx2, "Stock Splits"] = ss
-                quotes = quotes.drop(quotes.index[n - 1])
+                dropped_row = quotes.iloc[-1]
+                quotes = quotes.drop(quotes.index[-1])
 
-    return quotes
+    return quotes, dropped_row
 
 
 def safe_merge_dfs(df_main, df_sub, interval):


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 309daeb..d2250e5 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -15,7 +15,7 @@ import pandas as pd
 
 import unittest
 
-from yfinance.utils import is_valid_period_format
+from yfinance.utils import is_valid_period_format, _dts_in_same_interval
 
 
 class TestPandas(unittest.TestCase):
@@ -61,6 +61,118 @@ class TestUtils(unittest.TestCase):
         self.assertTrue(is_valid_period_format("999mo"))  # Large number valid
 
 
+class TestDateIntervalCheck(unittest.TestCase):
+    def test_same_day(self):
+        dt1 = pd.Timestamp("2024-10-15 10:00:00")
+        dt2 = pd.Timestamp("2024-10-15 14:30:00")
+        self.assertTrue(_dts_in_same_interval(dt1, dt2, "1d"))
+        
+    def test_different_days(self):
+        dt1 = pd.Timestamp("2024-10-15 10:00:00")
+        dt2 = pd.Timestamp("2024-10-16 09:00:00")
+        self.assertFalse(_dts_in_same_interval(dt1, dt2, "1d"))
+    
+    def test_same_week_mid_week(self):
+        # Wednesday and Friday in same week
+        dt1 = pd.Timestamp("2024-10-16")  # Wednesday
+        dt2 = pd.Timestamp("2024-10-18")  # Friday
+        self.assertTrue(_dts_in_same_interval(dt1, dt2, "1wk"))
+    
+    def test_different_weeks(self):
+        dt1 = pd.Timestamp("2024-10-14")  # Monday week 42
+        dt2 = pd.Timestamp("2024-10-21")  # Monday week 43
+        self.assertFalse(_dts_in_same_interval(dt1, dt2, "1wk"))
+    
+    def test_week_year_boundary(self):
+        # Week 52 of 2024 spans into 2025
+        dt1 = pd.Timestamp("2024-12-30")  # Monday in week 1 (ISO calendar)
+        dt2 = pd.Timestamp("2025-01-03")  # Friday in week 1 (ISO calendar)
+        self.assertTrue(_dts_in_same_interval(dt1, dt2, "1wk"))
+    
+    def test_same_month(self):
+        dt1 = pd.Timestamp("2024-10-01")
+        dt2 = pd.Timestamp("2024-10-31")
+        self.assertTrue(_dts_in_same_interval(dt1, dt2, "1mo"))
+    
+    def test_different_months(self):
+        dt1 = pd.Timestamp("2024-10-31")
+        dt2 = pd.Timestamp("2024-11-01")
+        self.assertFalse(_dts_in_same_interval(dt1, dt2, "1mo"))
+    
+    def test_month_year_boundary(self):
+        dt1 = pd.Timestamp("2024-12-15")
+        dt2 = pd.Timestamp("2025-01-15")
+        self.assertFalse(_dts_in_same_interval(dt1, dt2, "1mo"))
+
+    def test_standard_quarters(self):
+        q1_start = datetime(2023, 1, 1)
+        self.assertTrue(_dts_in_same_interval(q1_start, datetime(2023, 1, 15), '3mo'))
+        self.assertTrue(_dts_in_same_interval(q1_start, datetime(2023, 3, 31), '3mo'))
+        self.assertFalse(_dts_in_same_interval(q1_start, datetime(2023, 4, 1), '3mo'))
+        self.assertFalse(_dts_in_same_interval(q1_start, datetime(2022, 1, 15), '3mo'))  # Previous year
+        self.assertFalse(_dts_in_same_interval(q1_start, datetime(2024, 1, 15), '3mo'))  # Next year
+        
+        q2_start = datetime(2023, 4, 1)
+        self.assertTrue(_dts_in_same_interval(q2_start, datetime(2023, 5, 15), '3mo'))
+        self.assertTrue(_dts_in_same_interval(q2_start, datetime(2023, 6, 30), '3mo'))
+        self.assertFalse(_dts_in_same_interval(q2_start, datetime(2023, 7, 1), '3mo'))
+    
+    def test_nonstandard_quarters(self):
+        q1_start = datetime(2023, 2, 1)
+        # Same quarter
+        self.assertTrue(_dts_in_same_interval(q1_start, datetime(2023, 3, 1), '3mo'))
+        self.assertTrue(_dts_in_same_interval(q1_start, datetime(2023, 4, 25), '3mo'))
+        # Different quarters
+        self.assertFalse(_dts_in_same_interval(q1_start, datetime(2023, 1, 25), '3mo'))  # Before quarter start
+        self.assertFalse(_dts_in_same_interval(q1_start, datetime(2023, 6, 1), '3mo'))  # Start of next quarter
+        self.assertFalse(_dts_in_same_interval(q1_start, datetime(2023, 9, 1), '3mo'))  # Start of Q3
+        
+        q2_start = datetime(2023, 5, 1)
+        self.assertTrue(_dts_in_same_interval(q2_start, datetime(2023, 6, 1), '3mo'))
+        self.assertTrue(_dts_in_same_interval(q2_start, datetime(2023, 7, 25), '3mo'))
+        self.assertFalse(_dts_in_same_interval(q2_start, datetime(2023, 8, 1), '3mo'))
+    
+    def test_cross_year_quarters(self):
+        q4_start = datetime(2023, 11, 1)
+        
+        # Same quarter, different year
+        self.assertTrue(_dts_in_same_interval(q4_start, datetime(2023, 11, 15), '3mo'))
+        self.assertTrue(_dts_in_same_interval(q4_start, datetime(2024, 1, 15), '3mo'))
+        self.assertTrue(_dts_in_same_interval(q4_start, datetime(2024, 1, 25), '3mo'))
+        
+        # Different quarters
+        self.assertFalse(_dts_in_same_interval(q4_start, datetime(2024, 2, 1), '3mo'))  # Start of next quarter
+        self.assertFalse(_dts_in_same_interval(q4_start, datetime(2023, 10, 14), '3mo'))  # Before quarter start
+    
+    def test_hourly_interval(self):
+        dt1 = pd.Timestamp("2024-10-15 14:00:00")
+        dt2 = pd.Timestamp("2024-10-15 14:59:59")
+        self.assertTrue(_dts_in_same_interval(dt1, dt2, "1h"))
+        
+        dt3 = pd.Timestamp("2024-10-15 15:00:00")
+        self.assertFalse(_dts_in_same_interval(dt1, dt3, "1h"))
+    
+    def test_custom_intervals(self):
+        # Test 4 hour interval
+        dt1 = pd.Timestamp("2024-10-15 10:00:00")
+        dt2 = pd.Timestamp("2024-10-15 13:59:59")
+        self.assertTrue(_dts_in_same_interval(dt1, dt2, "4h"))
+        
+        dt3 = pd.Timestamp("2024-10-15 14:00:00")
+        self.assertFalse(_dts_in_same_interval(dt1, dt3, "4h"))
+        
+    def test_minute_intervals(self):
+        dt1 = pd.Timestamp("2024-10-15 10:30:00")
+        dt2 = pd.Timestamp("2024-10-15 10:30:45")
+        self.assertTrue(_dts_in_same_interval(dt1, dt2, "1min"))
+        
+        dt3 = pd.Timestamp("2024-10-15 10:31:00")
+        self.assertFalse(_dts_in_same_interval(dt1, dt3, "1min"))
+
+if __name__ == "__main__":
+    unittest.main()
+
+
 def suite():
     ts: TestSuite = unittest.TestSuite()
     ts.addTest(TestPandas("Test pandas"))