Package: histdatacom
histdata.com-toolsA Multi-threaded/Multi-Process command-line utility and python package that downloads currency exchange rates from Histdata.com. Imports to InfluxDB. Can be used in Jupyter Notebooks. Works on MacOS, Linux & Windows Systems.Requires Python3.10+NEW:Expanded API support!!!histdata.com-toolsDisclaimerUsageShow the Help and OptionsBasic UseAvailable FormatsCSV Dialect and Format SpecificationsDate Ranges'Start' & 'Now' KeywordsMultiple DatasetsCPU UtilizationImport to InfluxDBinfluxdb.yamlAPI - Other Scripts, Modules, & Jupyter SupportCLI AutomationJupyter and External ScriptsFull Script ExampleSetupTLDR for all platformsVanilla Python SetupVanilla MacOS and LinuxVanilla Windows PowershellAnaconda SetupAnaconda MacOS and LinuxAnaconda Windows using the Anaconda PromptData Table Installation OptionsRoadmapDisclaimer*I am in no way affiliated with histdata.com or its maintainers. Please use this application in a way that respects the hard work and resources of histdata.comIf you choose to use this tool, it isstronglysuggested that you head over tohttp://www.histdata.com/download-by-ftp/and sign up to help support their traffic costs.If you find this tool helpful and would like to support future development, I'm in need of caffeine, feel free tobuy me coffee!UsageNote #1The number one rule when using this tool is to beMOREspecific with your input to limit the size of your request.Note #2histdatacom is a very powerful tool and has the capability to fetch the entire repository housed on histdata.com. This isNEVERnecessary. If you are using this tool to fetch data for your favorite trading application, do not download data in all available formats.It is likely the default behavior will be modified from its current state to discourage unnecessarily large requests.*please submit feature requests and bug reports using this repository'sissue tracker.Show the help and optionshistdatacom -hhistdatacom -h
usage: histdatacom [-h] [-A] [-U] [--by BY] [--version] [-V] [-D] [-X] [-p PAIR [PAIR ...]] [-f FORMAT [FORMAT ...]] [-t TIMEFRAME [TIMEFRAME ...]] [-s START_YEARMONTH] [-e END_YEARMONTH] [-I] [-d] [-b BATCH_SIZE] [-c CPU_UTILIZATION]
                   [--data-directory DATA_DIRECTORY]

options:
  -h, --help            show this help message and exit

Mode:
  -V, --validate_urls   Check generated list of URLs as valid download locations
  -D, --download_data_archives
                        download specified pairs/formats/timeframe and create data files
  -X, --extract_csvs    histdata.com delivers zip files. Use the -X flag to extract them.

Config:
  -p PAIR [PAIR ...], --pairs PAIR [PAIR ...]
                        space separated currency pairs. e.g. -p eurusd usdjpy ...
  -f FORMAT [FORMAT ...], --formats FORMAT [FORMAT ...]
                        space separated formats. -f metatrader ascii ninjatrader metastock
  -t TIMEFRAME [TIMEFRAME ...], --timeframes TIMEFRAME [TIMEFRAME ...]
                        space separated Timeframes. -t tick-data-quotes 1-minute-bar-quotes
  -s START_YEARMONTH, --start_yearmonth START_YEARMONTH
                        set a start year and month for data. e.g. -s 2000-04 or -s 2015-00
  -e END_YEARMONTH, --end_yearmonth END_YEARMONTH
                        set a start year and month for data. e.g. -e 2020-00 or -e 2022-04

Influxdb:
  -I, --import_to_influxdb
                        import data to influxdb instance. Use influxdb.yaml to configure.
  -d, --delete_after_influx
                        delete data files after upload to influxdb
  -b BATCH_SIZE, --batch_size BATCH_SIZE
                        (integer) influxdb write_api batch size. defaults to 5000

System:
  -c CPU_UTILIZATION, --cpu_utilization CPU_UTILIZATION
                        "low", "medium", "high". High uses all available CPUs OR integer percent 1-200
  --data-directory DATA_DIRECTORY
                        Directory Used to save data. default is "./data/"

Info:
  -A, --available_remote_data
                        list data retrievable from histdata.com
  -U, --update_remote_data
                        update list of data retrievable from histdata.com
  --by BY               With -A, -U, to sort --by [pair_asc, pair_dsc, start_asc, start_dsc]
  --version             return current version of histdatacom.Basic UseDownload and extract the current month's available EURUSD data for metatrader 4/5into the default data directory ./datahistdatacom-peurusd-fmetatrader-snowinclude the-Dflag to download but NOT extract to csvhistdatacom-D-pusdcad-fmetastock-snowAvailable FormatsThe formats available are:metatradermetastockninjatraderexcelasciihistdata.com provides different resolutions of time
depending on the format.The following format/timeframe combinations are available:1-minute-bar-quotesall formatstick-data-quotesasciitick-last-quotesninjatradertick-bid-quotesninjatradertick-ask-quotesninjatraderCSV Dialect and Format SpecificationsFor Detailed specifications for the CSVs that the histdata.com repo provides seehistdata.com_data_specs.mdTo download 1-minute-bar-quotes for both metastock and excelhistdatacom-pusdjpy-fmetastockexcel-snowDate Rangesdate ranges are for year and month and can be specified in the following ways:[ -._]2022-04"2202 04"2202.042202_04to fetch a single year's data, leave out the monthnote: unless you're fetching data for the current year, tick data types will fetch 12 files for each month of the year, 1-minute-bar-quotes will fetch a single OHLC file with the whole year's data.histdatacom -p udxusd -f ascii -t tick-data-quotes -s 2011to fetch a single month's data, include a month, but do not use the-e, --end_yearmonthflagif you're requesting 1-minute-bar-quotes for any
year except the current year, you will receive the
the whole year's datathis example leaves out the-p --pairflag, and will
fetch data for all 66 available instrumentshistdatacom -f metatrader -s 2012-07Start&NowKeywordsyou may have noticed that two special year-month keywords existstartandnowstartmay only be used with the-s --start_yearmonthflag and the-e --end_yearmonthflagmustbe specified
to indicate a range of datahistdatacom -p audusd -f metatrader -s start -e 2008-12nowused alone will return the current year-monthwhen used with as-s nowit will return the most current month's datahistdatacom -p frxeur -f ninjatrader -s nowin the above example, no-t --timeframeflag was specified. This will return all time resolutions available for the specified format(s)nowwhen used with the-e --end_yearmonthflag is intended to be the end of a range. Rather, if the flags were to be-s 2019-04 -e nowthe request would return data from April 2019-04 to the present.histdatacom -p xagusd -f ascii -1-minute-bar-quotes -s 2019-04 -e nowMultiple Datasetsmultiple datasets can be requested in one commandthis example with use the-e --end_yearmonthflag to request a range of data for multiple instruments.note: Large requests like these are to be avoided. remember to sign up with histdata.com to help them pay for network costshistdatacom -p eurusd usdcad udxusd -f metatrader -s start -e 2017-04CPU UtilizationOne can set a cap on CPU Utilization with-c --cpu_utilizationavailable levels are,"low","medium","high"ORinteger percent 1-200
eg.-c 100is equal to-c highhistdatacom-cmedium-pudxusd-fmetatrader-s2015-04-e2016-04Import to InfluxDBTo import data to an influxdb instance, use the-I --import_to_influxdbflag along with aninfluxdb.yamlfile in the current working directory (where ever you are running the command from).ascii is the only format accepted for influxdb import.all histdata.com datetime data is in EST (Eastern Standard Time) with no adjustments for daylight savings.Influxdb does not adjust for timezone and all datetime data is recorded as UTC epoch timestamps (nano-seconds since midnight 00:00, January, 1st, 1970)this tool converts histdata.com ESTnoDST to UTC Epoch milli-second timestamps as part of the import-to-influx processhistdatacom -I -p eurusd -f ascii -t tick-data-quotes -s start -e nowinfluxdb.yaml# a sample influxdb.yaml file.influxdb:org:influx_orgbucket:data_bucketurl:influx_server_api_urltoken:influx_user_tokenDownload influxdb.yaml to your project's directorycurl"https://raw.githubusercontent.com/dmidlo/histdata.com-tools/main/influxdb.sample.yaml"--outputinfluxdb.yamlAPI - Other Scripts, Modules, & Jupyter Supporthistdatacom also has an API to allow developers and to integrate the package into their own projects.  It can be used in one of two ways; The first being a simple interface to automate CLI interaction. The second is as an interface to work with the data directly in a notebook environment like Jupyter Notebooks.CLI AutomationFirst import the required modulesimporthistdatacomfromhistdatacom.optionsimportOptionsCreate and Initialize a new options object to pass parameters to histdatacomoptions=Options()Configure for CLI automationTo automate the CLI, simply include one of the boolean behavior flags:options.validate_urls,options.download_data_archives,options.extract_csvs, andoptions.import_to_influxdbEach behavior flag implies the use of the preceding flags.histdatacom is an ETL pipeline (extract, transform, load) and each step depends on the preceding steps in the pipeline.For theCLI, the order of operations are:validate urlsdownload zip files from histdata.comextract the csv from the zip archivetransform the ESTnoDST datetime to UTC EpochANDupload to InfluxDB.# options.validate_urls = True# options.download_data_archives = True  # implies validateoptions.extract_csvs=True# implies validate and download# options.import_to_influxdb = True  # implies validate, download, and extractoptions.formats={"ascii"}options.timeframes={"tick-data-quotes"}options.pairs={"eurusd"}options.start_yearmonth="2021-04"options.end_yearmonth="now"options.cpu_utilization=100when a behavior flag is included,histdatacomassumes it is being used forCLIautomationexclusivelyand doesnotprovide a return value.at present, calling from another script or module is limited to using the__name__=="__main__"idiom.if__name__=="__main__":histdatacom(options)Jupyter may be used normallyhistdatacom(options)# (Jupyter)Jupyter and External ScriptsAs opposed to theCLIinterface, one may wish to load data from histdata.com and work with it interactively (e.g. in a Jupyter notebook), or as part of a larger pipeline.  To that end, histdatacom provides an option to specify a return type.return types can be:AdatatableFrameapandasdataframein Apachearrowin-memory formatto usepandasorarrowformats you must install the required packagespip install pandaspip install pyarrowAll datetime is returned as milliseconds since January 1, 1970 (midnight UTC/GMT)Import the required modulesimporthistdatacomfromhistdatacom.optionsimportOptionsInitialize a new options object to pass parameters to histdatacomoptions=Options()Jupyter & External Script Optionsoptions.api_return_type="pandas"# "datatable", "pandas", or "arrow"options.formats={"ascii"}# Must be {"ascii"}options.timeframes={"tick-data-quotes"}# can be tick-data-quotes or 1-minute-bar-quotesoptions.pairs={"eurusd"}options.start_yearmonth="2021-04"options.end_yearmonth="now"options.cpu_utilization="high"This example uses just one pair/instrument/symboleurusdand just one timeframetick-data-quotes.  When the api is called with this 'one-one` specificity, the api will directly return the requested data.Regardless of the specified start_yearmonth and end_yearmonth, the resultant data will be sorted and merged into a single dataset.Pass the options to histdatacom and assign the return to a variabledata=histdatacom(options)# (Jupyter)print(data)print(type(data))datetime      bid      ask  vol
0         1617253200478  1.17243  1.17244    0
1         1617253206261  1.17246  1.17248    0
2         1617253206362  1.17247  1.17249    0
3         1617253206946  1.17247  1.17250    0
4         1617253207121  1.17249  1.17250    0
...                 ...      ...      ...  ...
18648493  1650664783081  1.07968  1.08042    0
18648494  1650664783182  1.07968  1.08039    0
18648495  1650664790108  1.07964  1.08032    0
18648496  1650664790958  1.07947  1.08032    0
18648497  1650664794462  1.07947  1.08032    0

[18648498 rows x 4 columns]
<class 'pandas.core.frame.DataFrame'>When specifying more than one pair/symbol/instrument or timeframe, the api will return anlist of dictionarieswith references to the timeframe, pair, records used to create the data, and the merged data itself.options.api_return_type="pandas"options.formats={"ascii"}options.timeframes={"1-minute-bar-quotes"}options.pairs={"eurusd","usdcad"}options.start_yearmonth="2021-01"options.end_yearmonth="now"options.cpu_utilization="75"data=histdatacom(options)# (Jupyter)print(data)print(type(data))[
  {
    'timeframe': 'M1', 
    'pair': 'EURUSD', 
    'records': [<histdatacom.records.Record object ...>, ...],
    'data':    
                    datetime     open     high      low    close  vol
      0       1609711200000  1.22396  1.22396  1.22373  1.22395    0
      1       1609711260000  1.22387  1.22420  1.22385  1.22395    0
      2       1609711320000  1.22396  1.22398  1.22382  1.22382    0
      3       1609711380000  1.22383  1.22396  1.22376  1.22378    0
      4       1609711440000  1.22378  1.22385  1.22296  1.22347    0
      ...               ...      ...      ...      ...      ...  ...
      484172  1650664440000  1.07976  1.08014  1.07976  1.08014    0
      484173  1650664500000  1.08013  1.08021  1.07997  1.08000    0
      484174  1650664560000  1.08000  1.08000  1.07956  1.07968    0
      484175  1650664620000  1.07980  1.07980  1.07958  1.07968    0
      484176  1650664680000  1.07980  1.07986  1.07963  1.07963    0

      [484177 rows x 6 columns]
  }, 
  {
    'timeframe': 'M1', 
    'pair': 'USDCAD',
    'records': [<histdatacom.records.Record object ...>, ...],
    'data':                
                    datetime     open     high      low    close  vol
      0       1609711200000  1.27136  1.27201  1.27136  1.27201    0
      1       1609711260000  1.27207  1.27241  1.27207  1.27220    0
      2       1609711320000  1.27211  1.27219  1.27211  1.27219    0
      3       1609711380000  1.27212  1.27261  1.27212  1.27261    0
      4       1609711440000  1.27268  1.27268  1.27261  1.27261    0
      ...               ...      ...      ...      ...      ...  ...
      483946  1650664440000  1.27121  1.27132  1.27114  1.27131    0
      483947  1650664500000  1.27129  1.27137  1.27102  1.27106    0
      483948  1650664560000  1.27107  1.27114  1.27098  1.27101    0
      483949  1650664620000  1.27105  1.27105  1.27091  1.27091    0
      483950  1650664680000  1.27091  1.27097  1.27073  1.27097    0

      [483951 rows x 6 columns]
  }
]

<class 'list'>print(data[0]['timeframe'],data[0]['pair'])print(data[0]['data'])print(type(data[0]['data']))M1 EURUSD
               datetime     open     high      low    close  vol
0       20210103 170000  1.22396  1.22396  1.22373  1.22395    0
1       20210103 170100  1.22387  1.22420  1.22385  1.22395    0
2       20210103 170200  1.22396  1.22398  1.22382  1.22382    0
3       20210103 170300  1.22383  1.22396  1.22376  1.22378    0
4       20210103 170400  1.22378  1.22385  1.22296  1.22347    0
...                 ...      ...      ...      ...      ...  ...
484172  20220422 165400  1.07976  1.08014  1.07976  1.08014    0
484173  20220422 165500  1.08013  1.08021  1.07997  1.08000    0
484174  20220422 165600  1.08000  1.08000  1.07956  1.07968    0
484175  20220422 165700  1.07980  1.07980  1.07958  1.07968    0
484176  20220422 165800  1.07980  1.07986  1.07963  1.07963    0

[484177 rows x 6 columns]
<class 'pandas.core.frame.DataFrame'>at present, calling from another script or module is limited to using the__name__=="__main__"idiom.if__name__=="__main__":histdatacom(options)Jupyter may be used normallyhistdatacom(options)# (Jupyter)Full Script Exampleimporthistdatacomfromhistdatacom.optionsimportOptionsfromhistdatacom.fx_enumsimportPairsdefimport_pair_to_influx(pair,start,end):data_options=Options()data_options.import_to_influxdb=True# implies validate, download, and extractdata_options.delete_after_influx=Truedata_options.batch_size="2000"data_options.cpu_utilization="high"data_options.pairs={f"{pair}"}# histdata_and_oanda_intersect_symbsdata_options.start_yearmonth=f"{start}"data_options.end_yearmonth=f"{end}"data_options.formats={"ascii"}# Must be {"ascii"}data_options.timeframes={"tick-data-quotes"}# can be tick-data-quotes or 1-minute-bar-quoteshistdatacom(data_options)defget_available_range_data(pairs):range_options=Options()range_options.pairs=pairsrange_options.available_remote_data=Truerange_options.by="start_dsc"range_data=histdatacom(range_options)# (Jupyter)returnrange_datadefprint_one_datatable_frame(pair,start=None,end=None):options=Options()options.api_return_type="datatable"options.pairs={f"{pair}"}options.start_yearmonth="201501"options.formats={"ascii"}options.timeframes={"tick-data-quotes"}returnhistdatacom(options)defmain():histdata_symbs=Pairs.list_keys()# Oanda Symbols:oanda_symbs={"audcad","audchf","audhkd","audjpy","audsgd","audusd","cadhkd","cadjpy","cadsgd","chfhkd","chfjpy","euraud","eurcad","eurchf","eurgbp","eurhkd","eurjpy","eursgd","eurusd","gbpaud","gbpcad","gbpchf","gbphkd","gbpjpy","gbpsgd","gbpusd","hkdjpy","sgdchf","sgdhkd","sgdjpy","usdcad","usdchf","usdhkd","usdjpy","usdsgd","audnzd","cadchf","chfzar","eurczk","eurdkk","eurhuf","eurnok","eurnzd","eurpln","eursek","eurtry","eurzar","gbpnzd","gbppln","gbpzar","nzdcad","nzdchf","nzdhkd","nzdjpy","nzdsgd","nzdusd","tryjpy","usdcnh","usdczk","usddkk","usdhuf","usdmxn","usdnok","usdpln","usdsar","usdsek","usdthb","usdtry","usdzar","zarjpy"}histdata_and_oanda_intersect_symbs=histdata_symbs&oanda_symbspairs_data=get_available_range_data(histdata_and_oanda_intersect_symbs)forpairinpairs_data:start=pairs_data[pair]['start']end=pairs_data[pair]['end']import_pair_to_influx(pair,start,end)if__name__=='__main__':main()SetupTLDR for all platformsInstall the latest version of datatable*this is a temporary fix until the datatable team updates PyPi.See this issuefor more detailscheck out the section:Data Table Installation Optionsto either:install a build wheel from Datatable's Appveyor CI/CD pipeline, or;build from sourceInstall histdatacompipinstallhistdatacomto install latest development versionpipinstallgit+https://github.com/dmidlo/histdata.com-tools.gitVanilla MacOS and LinuxCreate a new project directory and change to itmkdirmyproject&&cdmyproject&&pwdCreate a Python Virtual Environment and activate itpython-mvenvvenv&&sourcevenv/bin/activateConfirm Python Path and Versionwhichpython&&python--versionBuild the latest version of datatablefollow the instructions fromInstall the latest version of datatableInstall the histdata.com-tools package from PyPipipinstallhistdatacomRunhistdatacomto view help message and Optionshistdatacom-hVanilla Windows PowershellLaunch a Powershell TerminalRun as Administrator (right-click on shortcut and click Run as Admin...)Make sure python3.10 is in your system's executable pathpython--versionshould be already set if you clicked the checkbox when installing python 3.10If not, you can run the following.you will need to relaunch powershell as admin.[Environment]::SetEnvironmentVariable("Path","$env:Path;C:\Program Files\Python310")Change the Execution Policy to UnrestrictedSet-ExecutionPolicyUnrestricted-ForceCreate a new directory and change to itNew-Item-Path".\"-Name"myproject"-ItemType"directory";Set-Location.\myproject\Create a Virtual Environment and activate itpython-mvenvvenv;.\venv\Scripts\Activate.ps1Confirm Path and VersionGet-Commandpython|selectSource;python--versionBuild the latest version of datatablefollow the instructions fromInstall the latest version of datatableInstall histdata.com-tools package from PyPipipinstallhistdatacomRunhistdatacomto view help messagehistdatacom-hAnaconda SetupAnaconda MacOS and LinuxCreate a Project Directory and Change to itmkdirmyproject&&cdmyproject&&pwdCreate aPython 3.10Anaconda environment withcondaand activate itcondacreate-npy310python=3.10&&condaactivatepy310Check Python Path and Versionwhichpython&&python--versionBuild the latest version of datatablefollow the instructions fromInstall the latest version of datatableInstall histdatacom package from PyPipipinstallhistdatacomRun histdatacom package to view help messagehistdatacom-hAnaconda Windows using the Anaconda PromptCreate a Directory and Change to itmkdirmyproject&&cdmyproject&&echo%cd%Create aPython 3.10Anaconda environment withcondaand activate itcondacreate-npy310python=3.10&&condaactivatepy310Check Python Path and Versionwherepython&&python--versionBuild the latest version of datatablefollow the instructions fromInstall the latest version of datatableInstall histdatacom package from PyPipipinstallhistdatacomRun histdatacom package to view help messagehistdatacom-hDatatable Installation OptionsInstall from AppveyorBuild wheels are pre-compiled versions of datatable, and would easily be the preferred route of installation while we wait for the datatable team to provide an official Python 3.10 package on PyPi.  The only drawback is documenting the procedure as the wheel's URL expires monthly thus this documentation could go out of date rather quickly...Activate Python Environment if you're using onerefer to theCreate a Python Virtual Environment and activate itsteps outlined for your platformVanilla MacOS and LinuxVanilla Windows PowershellAnaconda MacOS and LinuxAnaconda Windows using the Anaconda PromptGet the Build Wheel's URL for your platformTo find the latest build wheels for datatable, go to dataable'sAppveyor CI/CD Instance:Select the Platform you're installing for:Select"Artifacts"and right/option-click on the filename that containscp310. e.g.dist\datatable-1.1.0a2157-cp310-cp310-win_amd64.whlSelect"Copy Link Address"from your browser's context menu to copy the wheel's URLInstall datatable using pip with the wheel's URL from Appveyore.g.pip install {https://APPVEYOR DATATABLE BUILD WHEEL URL.whl}Build from SourceYou will need a C++ compiler installed to build datatable from sourceMacOS XCode Command Line ToolsForMacOS, runxcode-select --installfrom your terminal and confirm the prompts for download and installation of the xcode command-line tools.Windows MSVC C++ CompilerForWindows, you need to download and install theVisual Studio Community Editionand choose the optionDesktop Development with C++, then select install.Launch the Visual Studio command line environment (for Windows only)Open either apowershell,cmd, orAnaconda Promptterminalthe setup scripts for the VS CLI environments are located in the.\Common7\Tools\directory of your Visual Studio installation directorye.g."C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\"Run the VS CLI environment setup scriptforPowershell:PS> "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\Launch-VsDevShell.ps1"forCMDandAnaconda Prompt:> "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\LaunchDevCmd.bat"Tell the datatable setup where to find the MSVC C++ compilerforPowershell:PS> $env:DT_MSVC_PATH="$env:VSINSTALLDIR"+"VC\Tools\MSVC\"forCMDandAnaconda Prompt:set DT_MSVC_PATH=%VSINSTALLDIR%VC\Tools\MSVC\Return to Your Project's DirectoryThe Visual Studio command line environment setup scripts change your directory, you'll need to find your way back to your project's directory.  I like to use the variable%USERPROFILE%to save myself some typing:e.g.> cd %USERPROFILE%\Documents\projects\myprojectActivate Python Environment if you're using onerefer to theCreate a Python Virtual Environment and activate itsteps outlined for your platformVanilla MacOS and LinuxVanilla Windows PowershellAnaconda MacOS and LinuxAnaconda Windows using the Anaconda PromptInstall datatablepipinstallgit+https://github.com/h2oai/datatableRoadmapAdd Support for AnacondaImplement MyPy static typing checkingImplement UnitTesting with PyTestCreate Binary DistributionsSee about packaging for different operating systemsdeb/rpm packagingNuGet/ChocolateyMacPorts/Homebrewdocker imageCreate Down-sampling to Standard Candlestick TimeframesFix terminate on ctrl-c multiprocessing KeyboardInteruptLook at replacing beautifulsoup with html parserRefactor to make use of globals more readableadd -v -vv and -vvv flagsChange Record statuses to EnumAdd -S —set-status flagCreate a central place for exceptionsAdd the ability to import an order book to influxdbAdd a --reset-cache flag to reset all or specified year-month range