Package: pfeed
PFeed: Data Pipeline for Algo-Trading, Getting and Storing Real-Time and Historical Data Made Easy.PFeed (/piː fiːd/) is a data integration library tailored for algorithmic trading,
serving as an ETL (Extract, Transform, Load) data pipeline between raw data sources and traders,
helping them in creating a local data lake for quantitative research.PFeed allows traders to download historical, paper, and live data from various data sources, both free and paid,
and stores them into a local data lake usingMinIO.It is designed to be used alongsidePFund— A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready. Supports Vectorized and Event-Driven Backtesting, Paper and Live Trading.Table of ContentsProject StatusMissionCore FeaturesInstallationQuick StartDownload Historical Data on Command LineDownload Historical Data in PythonSupported Data SourcesRelated ProjectsProject StatusCaution: PFeed is at a VERY EARLY stage, use it at your own risk.PFeed is currently under active development, the framework design will be prioritized first over
stability and scalability.Please note that the available version is adevversion, not astableone.You are encouraged to play with thedevversion, but only use it when astableversion is released.PFeed for the time being only supportsBybitand Yahoo Finance for testing purpose.MissionAlgo-trading has always been a complicated task due to the multitude of components and procedures involved.Data collection and processing is probably the most mundane and yet critical part of it, as all results and findings
are derived from the data.However, preparing this data for use is not quick and easy. For example, sometimes even when the data is available (e.g.Bybit data), it is often in raw form and requires some cleaning.PFeed's mission is tofree traders from the tedious data workby providing cleaned data in a standard format that is ready for use, making them significantly faster to get to the analysis and strategy development phase.Core FeaturesUnified approach for interacting with various data sources, obtaining historical and real-time dataETL data pipline for transforming raw data and storing it inMinIO(optional)UtilizesRayfor parallel data downloadingIntegrates withPrefectto control data flowsListens to PFund's trade engine and adds trade history to a local databaseTimescaledb(optional)InstallationUsingPoetry(Recommended)poetryaddpfeedUsing PippipinstallpfeedQuick StartDownload Historical Data on the Command Line Interface (CLI)# download datapfeeddownload-sbybit-pBTC_USDT_PERP# enable debug modepfeeddownload-sbybit-pBTC_USDT_PERP--debug# list the current config:pfeedconfig--list# change the data storage location to your local project's 'data' folder:pfeedconfig--data-path./data# for more commands:pfeed--helpDownload Historical Data in Pythonfrompfeedimportbybitbybit.download(pdts=['BTC_USDT_PERP'])Run PFeed's docker-compose.yml# same as 'docker-compose', only difference is it has pointed to pfeed's docker-compose.yml filepfeeddocker-compose[command]# e.g. start servicespfeeddocker-composeup-d# e.g. stop servicespfeeddocker-composedownSupported Data SourcesData SourceGet Historical DataDownload Historical DataGet Live/Paper DataStream Live/Paper DataYahoo Finance🟢⚪⚪⚪Bybit🟢🟢🟡🔴*Interactive Brokers (IB)🔴⚪🔴🔴*FirstRate Data🔴🔴⚪⚪Binance🔴🔴🔴🔴OKX🔴🔴🔴🔴🟢 = finished🟡 = in progress🔴 = todo⚪ = not applicable* = paid dataget data = store it in memory for python to usedownload data = store it in local machine for later research
stream data = listen to real-time data and store it optionallyRelated ProjectsPFund— A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready. Supports Vectorized and Event-Driven Backtesting, Paper and Live Trading