Package: aioTrends
aioTrendsIntro.This is a project for asynchronously obtaining data from google trends in an efficient way. Inspired bypytrends, I am developing this project based on a asynchronous framework, asyncio, and a related module,aiohttp.The logic behind this project is to firstly build a cookies pool, then obtain and store the tokenized queries (wrapped inside the widgets) in another pool, and lastly retreive the data with widgets from the widget pool.Only data of interest over time is tested and avaiable now.Pros & ConsProsSaving time~ By employing the asynchronous framework, the programme will deal with other requests while waiting for responses from Google Trends, saving the waiting time.Saving repeated requests~ Suffering from broken connections and being tired of restarting the requests process? This programme separates the whole process into (1) building a cookies pool, (2) building a widgets pool and (3) retrieving data. The programme can be started from either sub-stage, avoiding sending repeated requests.Unlimited tasks amount~ Tons of queries? The programme will handle that for you automatically.ConsHeavily relying on proxies~ When running on a large amount of queries, proxies would be required for successfully catching responses. In this context, a small amount of rotating proxies or a large amount of static proxies would be required.Only timeseries interest data is avaiable now~ Will test others in the future.Requirementspython >= 3.10aiohttpaiofilesnumpypandasFilesSettings can be customized by amending the settings.json under the foler settings.An example input of queries is given under the data folder.An example of proxies file is given under the proxies folder.The file userAgents.json is fromSaid-Ait-Driss.Before StartI. Initial stageInstallPython version at least 3.10if you don't have one, I use python 3.11 in this exampleInstall package via pip command line. (On macOS's terminal or WindowsOS's CMD)pip install aioTrends
pip install virtualenvCreate a virtual environment, named as atenv, for running python3.11 without affecting your other setups.where python3.11copy the path to python 3.11 and replace below pathvirtualenv -p /path/to/python3.11 atenvActivate the virtual environmentOn Windows:atenv\Scripts\activateOn macOS and Linux:source atenv/bin/activateInstall aioTrends ********the package must be installed under the environment of python 3.10+pip install aioTrendsChecking if installed properly. The programme will creat folders, please follow the instructions given by the programme.cd path/to/your/working/path
python
import aioTrends as atAmend the settings.json under the folder 'settings'.Paste proxies to the proxies.txt under the folder 'proxies'.Get userAgents.json file fromSaid-Ait-Drissand past it under the folder 'settings'.Getting StartedII. Setup a queries fileimportpickleqrys={0:{'keywords':['AAPL'],'periods':'2007-01-01 2007-08-31','freq':'D'},1:{'keywords':['AMZN'],'periods':'all','freq':'M'},2:{'keywords':['AAPL','AMZN'],'periods':'all','freq':'M'},...10000:{'keywords':['MSFT'],'periods':'2004-01-01 2022-12-31','freq':'M'}}pickle.dump(qrys,open('./data/qrys.pkl','wb'))Alternatively, functionformQuerieswould form the query dataset based on the list of keywords you give.fromaioTrendsimportformQueriesfromdatetimeimportdateimportpickleqrys=formQueries(keywords=['AMZN','MSFN'],start='2004-01-01',end=date.today(),freq='D')pickle.dump(qrys,open('./data/qrys.pkl','wb'))III. Create a py script named as example.pyimportaioTrendsasat#Step 0: Set the log file. Other settings can be customized by amending the settings.json under the folder settings.at.setLog('./data/hello.log')#Step 1: collect 1000 cookies with 100 cocurrent tasks. Cocurrent tasks amount can be customized.at.CookeisPool(100).run(1000)#Step 2: get widgets with 100 cocurrent tasks. Cocurrent tasks can be customized.at.WidgetsPool(100).run()#Step 3: get data with 100 cocurrent tasks. Cocurrent tasks can be customized.at.DataInterestOverTime(100).run()Alternatively, you can use below one line for forming queries and getting daily scaled data or monthly data.importaioTrendsasatfromdatetimeimportdateqry_list=['AMZN','AAPL','MSFT']# running 50 cocurrent tasksataio=at.Aio(50)df=ataio.getScaledDailyData(keywords=qry_list,# the query keyword listfilename='test.csv',# json and pickle are both supportedstart='2004-01-01',# both datetime and str are supportedend=date.today())fig=df.plot(figsize=(16,8),title='TEST_SCALED_DAILY_DATA').get_figure()fig.savefig('test_scaled_daily_data.png')df_m=ataio.getMonthlyData(keywords=qry_list,start='2004-01-01',end='2022-12-31')fig=df_m.plot(figsize=(16,8),title='TEST_MONTHLY_DATA').get_figure()fig.savefig('test_monthly_data.png')IV. Run the above example.py file on your terminal or cmd (The code need to be running under the python 3.10+ environment)python example.py