Repository: GeneralMills/pytrends

Issue:
Example in readme for extracting topic doesn't work
See [notebook](https://colab.research.google.com/drive/13tOcn6AyUGMBqRXcm6hcsVDT8qpEUOld#scrollTo=d057Oqr7GIE2), which follows the topic example in the readme:

```
from pytrends.request import TrendReq

pytrend = TrendReq(hl='en-US', tz=360)
pytrend.build_payload(kw_list=['"%2Fm%2F025rw19"'],
                      cat=0, timeframe='today 5-y', geo='', gprop='')
pytrend.interest_over_time()
```
This returns nothing. Replacing `%2F` with `/` works though:
```
pytrend.build_payload(kw_list=['/m/025rw19'],
                      cat=0, timeframe='today 5-y', geo='', gprop='')
pytrend.interest_over_time()
```
```
/m/025rw19 | isPartial
-- | --

61 | False
59 | False
...
```

Comments:

Patch:
diff --git a/README.md b/README.md
index 7515d36..d7cf567 100644
--- a/README.md
+++ b/README.md
@@ -7,10 +7,10 @@ Unofficial API for Google Trends
 Allows simple interface for automating downloading of reports from Google Trends. 
 Only good until Google changes their backend again :-P. When that happens feel free to contribute!
 
-**Looking for maintainers!**
+**Looking for maintainers!** Please open an issue with a method of contacting you if you're interested.
 
 
-## Table of contens
+## Table of Contents
 
 * [Installation](#installation)
 
@@ -18,7 +18,7 @@ Only good until Google changes their backend again :-P. When that happens feel f
 
   * [API Methods](#api-methods)
 
-  * [Common API parameters](#common-api-parameters)
+  * [Common API Parameters](#common-api-parameters)
 
     * [Interest Over Time](#interest-over-time)
     * [Historical Hourly Interest](#historical-hourly-interest)
@@ -26,6 +26,7 @@ Only good until Google changes their backend again :-P. When that happens feel f
     * [Related Topics](#related-topics)
     * [Related Queries](#related-queries)
     * [Trending Searches](#trending-searches)
+    * [Realtime Search Trends](#realtime-search-trends)
     * [Top Charts](#top-charts)
     * [Suggestions](#suggestions)
 
@@ -136,13 +137,13 @@ Many API methods use the following:
       - For example ```"iron"``` will have a drop down of ```"Iron Chemical Element, Iron Cross, Iron Man, etc"```.
       - Find the encoded topic by using the get_suggestions() function and choose the most relevant one for you.
       - For example: ```https://www.google.com/trends/explore#q=%2Fm%2F025rw19&cmpt=q```
-      - ```"%2Fm%2F025rw19"``` is the topic "Iron Chemical Element" to use this with pytrends
+      - ```"/m/025rw19"``` is the topic "Iron Chemical Element" to use this with pytrends
       - You can also use `pytrends.suggestions()` to automate this.
 
 * `cat`
 
   - Category to narrow results
-  - Find available cateogies by inspecting the url when manually using Google Trends. The category starts after ```cat=``` and ends before the next ```&``` or view this [wiki page containing all available categories](https://github.com/pat310/google-trends-api/wiki/Google-Trends-Categories)
+  - Find available categories by inspecting the url when manually using Google Trends. The category starts after ```cat=``` and ends before the next ```&``` or view this [wiki page containing all available categories](https://github.com/pat310/google-trends-api/wiki/Google-Trends-Categories)
   - For example: ```"https://www.google.com/trends/explore#q=pizza&cat=71"```
   - ```'71'``` is the category
   - Defaults to no category
@@ -152,7 +153,7 @@ Many API methods use the following:
   - Two letter country abbreviation
   - For example United States is ```'US'```
   - Defaults to World
-  - More detail available for States/Provinces by specifying additonal abbreviations
+  - More detail available for States/Provinces by specifying additional abbreviations
   - For example: Alabama would be ```'US-AL'```
   - For example: England would be ```'GB-ENG'```
 
@@ -278,6 +279,15 @@ Returns pandas.DataFrame
 
 <sub><sup>[back to top](#trending_searches)</sub></sup>
 
+### Realtime Search Trends
+
+	pytrends.realtime_trending_searches(pn='US') # realtime search trends for United States
+	pytrends.realtime_trending_searches(pn='IN') # India
+
+Returns pandas.DataFrame
+
+<sub><sup>[back to top](#realtime-search-trends)</sub></sup>
+
 ### Top Charts
 
     pytrends.top_charts(date, hl='en-US', tz=300, geo='GLOBAL')
@@ -325,7 +335,7 @@ Returns dictionary
 * Google may change aggregation level for items with very large or very small search volume
 * Rate Limit is not publicly known, let me know if you have a consistent estimate
   * One user reports that 1,400 sequential requests of a 4 hours timeframe got them to the limit. (Replicated on 2 networks)
-  * It has been tested, and 60 seconds of sleep between requests (successful or not) is the correct amount once you reach the limit.
+  * It has been tested, and 60 seconds of sleep between requests (successful or not) appears to be the correct amount once you reach the limit.
 * For certain configurations the dependency lib certifi requires the environment variable REQUESTS_CA_BUNDLE to be explicitly set and exported. This variable must contain the path where the ca-certificates are saved or a SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] error is given at runtime. 
 
 # Credits
diff --git a/examples/example.py b/examples/example.py
index 0fb84ae..41a49a5 100644
--- a/examples/example.py
+++ b/examples/example.py
@@ -33,3 +33,8 @@ print(top_charts_df.head())
 # Get Google Keyword Suggestions
 suggestions_dict = pytrend.suggestions(keyword='pizza')
 print(suggestions_dict)
+
+# Get Google Realtime Search Trends
+
+realtime_searches = pytrend.realtime_trending_searches(pn='IN')
+print(realtime_searches.head())
diff --git a/pytrends/request.py b/pytrends/request.py
index 5a333e6..6cb9f8d 100644
--- a/pytrends/request.py
+++ b/pytrends/request.py
@@ -30,6 +30,7 @@ class TrendReq(object):
     SUGGESTIONS_URL = 'https://trends.google.com/trends/api/autocomplete/'
     CATEGORIES_URL = 'https://trends.google.com/trends/api/explore/pickers/category'
     TODAY_SEARCHES_URL = 'https://trends.google.com/trends/api/dailytrends'
+    REALTIME_TRENDING_SEARCHES_URL = 'https://trends.google.com/trends/api/realtimetrends'
     ERROR_CODES = (500, 502, 504, 429)
 
     def __init__(self, hl='en-US', tz=360, geo='', timeout=(2, 5), proxies='',
@@ -324,8 +325,11 @@ class TrendReq(object):
         result_dict = dict()
         for request_json in self.related_topics_widget_list:
             # ensure we know which keyword we are looking at rather than relying on order
-            kw = request_json['request']['restriction'][
-                'complexKeywordsRestriction']['keyword'][0]['value']
+            try:
+                kw = request_json['request']['restriction'][
+                    'complexKeywordsRestriction']['keyword'][0]['value']
+            except KeyError:
+                kw = ''
             # convert to string as requests will mangle
             related_payload['req'] = json.dumps(request_json['request'])
             related_payload['token'] = request_json['token']
@@ -373,8 +377,11 @@ class TrendReq(object):
         result_dict = dict()
         for request_json in self.related_queries_widget_list:
             # ensure we know which keyword we are looking at rather than relying on order
-            kw = request_json['request']['restriction'][
-                'complexKeywordsRestriction']['keyword'][0]['value']
+            try:
+                kw = request_json['request']['restriction'][
+                    'complexKeywordsRestriction']['keyword'][0]['value']
+            except KeyError:
+                kw = ''
             # convert to string as requests will mangle
             related_payload['req'] = json.dumps(request_json['request'])
             related_payload['token'] = request_json['token']
@@ -441,6 +448,44 @@ class TrendReq(object):
         result_df = pd.concat([result_df, sub_df])
         return result_df.iloc[:, -1]
 
+    def realtime_trending_searches(self, pn='US', cat='all', count =300):
+        """Request data from Google Realtime Search Trends section and returns a dataframe"""
+        # Don't know what some of the params mean here, followed the nodejs library
+        # https://github.com/pat310/google-trends-api/ 's implemenration
+
+
+        #sort: api accepts only 0 as the value, optional parameter
+
+        # ri: number of trending stories IDs returned,
+        # max value of ri supported is 300, based on emperical evidence
+
+        ri_value = 300
+        if count < ri_value:
+            ri_value = count
+
+        # rs : don't know what is does but it's max value is never more than the ri_value based on emperical evidence
+        # max value of ri supported is 200, based on emperical evidence
+        rs_value = 200
+        if count < rs_value:
+            rs_value = count-1
+
+        forms = {'ns': 15, 'geo': pn, 'tz': '300', 'hl': 'en-US', 'cat': cat, 'fi' : '0', 'fs' : '0', 'ri' : ri_value, 'rs' : rs_value, 'sort' : 0}
+        req_json = self._get_data(
+            url=TrendReq.REALTIME_TRENDING_SEARCHES_URL,
+            method=TrendReq.GET_METHOD,
+            trim_chars=5,
+            params=forms
+        )['storySummaries']['trendingStories']
+
+        # parse the returned json
+        wanted_keys = ["entityNames", "title"]
+
+        final_json = [{ key: ts[key] for key in ts.keys() if key in wanted_keys} for ts in req_json ]
+
+        result_df = pd.DataFrame(final_json)
+
+        return result_df
+
     def top_charts(self, date, hl='en-US', tz=300, geo='GLOBAL'):
         """Request data from Google's Top Charts section and return a dataframe"""
 
@@ -501,7 +546,7 @@ class TrendReq(object):
     def get_historical_interest(self, keywords, year_start=2018, month_start=1,
                                 day_start=1, hour_start=0, year_end=2018,
                                 month_end=2, day_end=1, hour_end=0, cat=0,
-                                geo='', gprop='', sleep=0):
+                                geo='', gprop='', sleep=0, frequency='hourly'):
         """Gets historical hourly data for interest by chunking requests to 1 week at a time (which is what Google allows)"""
 
         # construct datetime objects - raises ValueError if invalid parameters
@@ -509,8 +554,17 @@ class TrendReq(object):
                                                    day_start, hour_start)
         end_date = datetime(year_end, month_end, day_end, hour_end)
 
-        # the timeframe has to be in 1 week intervals or Google will reject it
-        delta = timedelta(days=7)
+        # Timedeltas:
+        # 7 days for hourly
+        # ~250 days for daily (270 seems to be max but sometimes breaks?)
+        # For weekly can pull any date range so no method required here
+        
+        if frequency == 'hourly':
+            delta = timedelta(days=7)
+        elif frequency == 'daily':
+            delta = timedelta(days=250)
+        else:
+            raise(ValueError('Frequency must be hourly or daily'))
 
         df = pd.DataFrame()
 
@@ -518,10 +572,14 @@ class TrendReq(object):
         date_iterator += delta
 
         while True:
-            # format date to comply with API call
+            # format date to comply with API call (different for hourly/daily)
 
-            start_date_str = start_date.strftime('%Y-%m-%dT%H')
-            date_iterator_str = date_iterator.strftime('%Y-%m-%dT%H')
+            if frequency == 'hourly':
+                start_date_str = start_date.strftime('%Y-%m-%dT%H')
+                date_iterator_str = date_iterator.strftime('%Y-%m-%dT%H')
+            elif frequency == 'daily':
+                start_date_str = start_date.strftime('%Y-%m-%d')
+                date_iterator_str = date_iterator.strftime('%Y-%m-%d')
 
             tf = start_date_str + ' ' + date_iterator_str
 
@@ -537,10 +595,13 @@ class TrendReq(object):
             date_iterator += delta
 
             if (date_iterator > end_date):
-                # Run for 7 more days to get remaining data that would have been truncated if we stopped now
-                # This is needed because google requires 7 days yet we may end up with a week result less than a full week
-                start_date_str = start_date.strftime('%Y-%m-%dT%H')
-                date_iterator_str = date_iterator.strftime('%Y-%m-%dT%H')
+                # Run more days to get remaining data that would have been truncated if we stopped now
+                if frequency == 'hourly':
+                    start_date_str = start_date.strftime('%Y-%m-%dT%H')
+                    date_iterator_str = date_iterator.strftime('%Y-%m-%dT%H')
+                elif frequency == 'daily':
+                    start_date_str = start_date.strftime('%Y-%m-%d')
+                    date_iterator_str = date_iterator.strftime('%Y-%m-%d')
 
                 tf = start_date_str + ' ' + date_iterator_str
 


Test patch:
diff --git a/pytrends/test_trendReq.py b/pytrends/test_trendReq.py
index 8c3efd3..3e774b9 100644
--- a/pytrends/test_trendReq.py
+++ b/pytrends/test_trendReq.py
@@ -75,6 +75,11 @@ class TestTrendReq(TestCase):
         pytrend.build_payload(kw_list=['pizza', 'bagel'])
         self.assertIsNotNone(pytrend.trending_searches())
 
+    def test_realtime_trending_searches(self):
+        pytrend = TrendReq()
+        pytrend.build_payload(kw_list=['pizza', 'bagel'])
+        self.assertIsNotNone(pytrend.realtime_trending_searches(pn='IN'))
+
     def test_top_charts(self):
         pytrend = TrendReq()
         pytrend.build_payload(kw_list=['pizza', 'bagel'])