Package: pyqlib
:newspaper:What's NEW!:sparkling_heart:Recent released featuresFeatureStatusKRNN and Sandwich models:chart_with_upwards_trend:Releasedon May 26, 2023Release Qlib v0.9.0:octocat:Releasedon Dec 9, 2022RL Learning Framework:hammer: :chart_with_upwards_trend: Released on Nov 10, 2022.#1332,#1322,#1316,#1299,#1263,#1244,#1169,#1125,#1076HIST and IGMTF models:chart_with_upwards_trend:Releasedon Apr 10, 2022Qlibnotebook tutorial📖Releasedon Apr 7, 2022Ibovespa index data:rice:Releasedon Apr 6, 2022Point-in-Time database:hammer:Releasedon Mar 10, 2022Arctic Provider Backend & Orderbook data example:hammer:Releasedon Jan 17, 2022Meta-Learning-based framework & DDG-DA:chart_with_upwards_trend:  :hammer:Releasedon Jan 10, 2022Planning-based portfolio optimization:hammer:Releasedon Dec 28, 2021Release Qlib v0.8.0:octocat:Releasedon Dec 8, 2021ADD model:chart_with_upwards_trend:Releasedon Nov 22, 2021ADARNN  model:chart_with_upwards_trend:Releasedon Nov 14, 2021TCN  model:chart_with_upwards_trend:Releasedon Nov 4, 2021Nested Decision Framework:hammer:Releasedon Oct 1, 2021.ExampleandDocTemporal Routing Adaptor (TRA):chart_with_upwards_trend:Releasedon July 30, 2021Transformer & Localformer:chart_with_upwards_trend:Releasedon July 22, 2021Release Qlib v0.7.0:octocat:Releasedon July 12, 2021TCTS Model:chart_with_upwards_trend:Releasedon July 1, 2021Online serving and automatic model rolling:hammer:Releasedon May 17, 2021DoubleEnsemble Model:chart_with_upwards_trend:Releasedon Mar 2, 2021High-frequency data processing example:hammer:Releasedon Feb 5, 2021High-frequency trading example:chart_with_upwards_trend:Part of code releasedon Jan 28, 2021High-frequency data(1min):rice:Releasedon Jan 27, 2021Tabnet Model:chart_with_upwards_trend:Releasedon Jan 22, 2021Features released before 2021 are not listed here.Qlib is an open-source, AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning.An increasing number of SOTA Quant research works/papers in diverse paradigms are being released in Qlib to collaboratively solve key challenges in quantitative investment. For example, 1) using supervised learning to mine the market's complex non-linear patterns from rich and heterogeneous financial data, 2) modeling the dynamic nature of the financial market using adaptive concept drift technology, and 3) using reinforcement learning to model continuous investment decisions and assist investors in optimizing their trading strategies.It contains the full ML pipeline of data processing, model training, back-testing; and covers the entire chain of quantitative investment: alpha seeking, risk modeling, portfolio optimization, and order execution.
For more details, please refer to our paper"Qlib: An AI-oriented Quantitative Investment Platform".Frameworks, Tutorial, Data & DevOpsMain Challenges & Solutions in Quant ResearchPlansFramework of QlibQuick StartInstallationData PreparationAuto Quant Research WorkflowBuilding Customized Quant Research Workflow by CodeQuant Dataset ZooLearning FrameworkMore About QlibOffline Mode and Online ModePerformance of Qlib Data ServerRelated ReportsContact UsContributingMain Challenges & Solutions in Quant ResearchForecasting: Finding Valuable Signals/PatternsQuant Model (Paper) ZooRun a Single ModelRun Multiple ModelsAdapting to Market DynamicsReinforcement Learning: modeling continuous decisionsPlansNew features under development(order by estimated release time).
Your feedbacks about the features are very important.Framework of QlibThe high-level framework of Qlib can be found above(users can find thedetailed frameworkof Qlib's design when getting into nitty gritty).
The components are designed as loose-coupled modules, and each component could be used stand-alone.Qlib provides a strong infrastructure to support Quant research.Datais always an important part.
A strong learning framework is designed to support diverse learning paradigms (e.g.reinforcement learning,supervised learning) and patterns at different levels(e.g.market dynamic modeling).
By modeling the market,trading strategieswill generate trade decisions that will be executed. Multiple trading strategies and executors in different levels or granularities can benested to be optimized and run together.
At last, a comprehensiveanalysiswill be provided and the model can beserved onlinein a low cost.Quick StartThis quick start guide tries to demonstrateIt's very easy to build a complete Quant research workflow and try your ideas withQlib.Though withpublic dataandsimple models, machine learning technologieswork very wellin practical Quant investment.Here is a quickdemoshows how to installQlib, and run LightGBM withqrun.But, please make sure you have already prepared the data following theinstruction.InstallationThis table demonstrates the supported Python version ofQlib:install with pipinstall from sourceplotPython 3.7:heavy_check_mark::heavy_check_mark::heavy_check_mark:Python 3.8:heavy_check_mark::heavy_check_mark::heavy_check_mark:Python 3.9:x::heavy_check_mark::x:Note:Condais suggested for managing your Python environment.Please pay attention that installing cython in Python 3.6 will raise some error when installingQlibfrom source. If users use Python 3.6 on their machines, it is recommended toupgradePython to version 3.7 or useconda's Python to installQlibfrom source.For Python 3.9,Qlibsupports running workflows such as training models, doing backtest and plot most of the related figures (those included innotebook). However, plotting for themodel performanceis not supported for now and we will fix this when the dependent packages are upgraded in the future.QlibRequirestablespackage,hdf5in tables does not support python3.9.Install with pipUsers can easily installQlibby pip according to the following command.pipinstallpyqlibNote: pip will install the latest stable qlib. However, the main branch of qlib is in active development. If you want to test the latest scripts or functions in the main branch. Please install qlib with the methods below.Install from sourceAlso, users can install the latest dev versionQlibby the source code according to the following steps:Before installingQlibfrom source, users need to install some dependencies:pipinstallnumpy
pipinstall--upgradecythonClone the repository and installQlibas follows.gitclonehttps://github.com/microsoft/qlib.git&&cdqlib
pipinstall.Note:  You can install Qlib withpython setup.py installas well. But it is not the recommended approach. It will skippipand cause obscure problems. For example,onlythe commandpip install .canoverwrite the stable version installed bypip install pyqlib, while the commandpython setup.py installcan't.Tips: If you fail to installQlibor run the examples in your environment,  comparing your steps and theCI workflowmay help you find the problem.Data PreparationLoad and prepare data by running the following code:Get with module# get 1d datapython-mqlib.run.get_dataqlib_data--target_dir~/.qlib/qlib_data/cn_data--regioncn# get 1min datapython-mqlib.run.get_dataqlib_data--target_dir~/.qlib/qlib_data/cn_data_1min--regioncn--interval1minGet from source# get 1d datapythonscripts/get_data.pyqlib_data--target_dir~/.qlib/qlib_data/cn_data--regioncn# get 1min datapythonscripts/get_data.pyqlib_data--target_dir~/.qlib/qlib_data/cn_data_1min--regioncn--interval1minThis dataset is created by public data collected bycrawler scripts, which have been released in
the same repository.
Users could create the same dataset with it.Description of datasetPlease payATTENTIONthat the data is collected fromYahoo Finance, and the data might not be perfect.
We recommend users to prepare their own data if they have a high-quality dataset. For more information, users can refer to therelated document.Automatic update of daily frequency data (from yahoo finance)This step isOptionalif users only want to try their models and strategies on history data.It is recommended that users update the data manually once (--trading_date 2021-05-25) and then set it to update automatically.NOTE: Users can't incrementally  update data based on the offline data provided by Qlib(some fields are removed to reduce the data size). Users should useyahoo collectorto download Yahoo data from scratch and then incrementally update it.For more information, please refer to:yahoo collectorAutomatic update of data to the "qlib" directory each trading day(Linux)usecrontab:crontab -eset up timed tasks:* * * * 1-5 python <script path> update_data_to_bin --qlib_data_1d_dir <user data dir>script path:scripts/data_collector/yahoo/collector.pyManual update of datapython scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --trading_date <start date> --end_date <end date>trading_date: start of trading dayend_date: end of trading day(not included)Auto Quant Research WorkflowQlib provides a tool namedqrunto run the whole workflow automatically (including building dataset, training models, backtest and evaluation). You can start an auto quant research workflow and have a graphical reports analysis according to the following steps:Quant Research Workflow: Runqrunwith lightgbm workflow config (workflow_config_lightgbm_Alpha158.yamlas following.cdexamples# Avoid running program under the directory contains `qlib`qrunbenchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yamlIf users want to useqrununder debug mode, please use the following command:python-mpdbqlib/workflow/cli.pyexamples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yamlThe result ofqrunis as follows, please refer toIntraday Tradingfor more details about the result.'The following are analysis results of the excess return without cost.'risk
mean0.000708
std0.005626
annualized_return0.178316
information_ratio1.996555
max_drawdown-0.081806'The following are analysis results of the excess return with cost.'risk
mean0.000512
std0.005626
annualized_return0.128982
information_ratio1.444287
max_drawdown-0.091078Here are detailed documents forqrunandworkflow.Graphical Reports Analysis: Runexamples/workflow_by_code.ipynbwithjupyter notebookto get graphical reportsForecasting signal (model prediction) analysisCumulative Return of groupsReturn distributionInformation Coefficient (IC)Auto Correlation of forecasting signal (model prediction)Portfolio analysisBacktest returnExplanationof above resultsBuilding Customized Quant Research Workflow by CodeThe automatic workflow may not suit the research workflow of all Quant researchers. To support a flexible Quant research workflow, Qlib also provides a modularized interface to allow researchers to build their own workflow by code.Hereis a demo for customized Quant research workflow by code.Main Challenges & Solutions in Quant ResearchQuant investment is an very unique scenario with lots of key challenges to be solved.
Currently, Qlib provides some solutions for several of them.Forecasting: Finding Valuable Signals/PatternsAccurate forecasting of the stock price trend is a very important part to construct profitable portfolios.
However, huge amount of data with various formats in the financial market which make it challenging to build forecasting models.An increasing number of SOTA Quant research works/papers, which focus on building forecasting models to mine valuable signals/patterns in complex financial data, are released inQlibQuant Model (Paper) ZooHere is a list of models built onQlib.GBDT based on XGBoost (Tianqi Chen, et al. KDD 2016)GBDT based on LightGBM (Guolin Ke, et al. NIPS 2017)GBDT based on Catboost (Liudmila Prokhorenkova, et al. NIPS 2018)MLP based on pytorchLSTM based on pytorch (Sepp Hochreiter, et al. Neural computation 1997)GRU based on pytorch (Kyunghyun Cho, et al. 2014)ALSTM based on pytorch (Yao Qin, et al. IJCAI 2017)GATs based on pytorch (Petar Velickovic, et al. 2017)SFM based on pytorch (Liheng Zhang, et al. KDD 2017)TFT based on tensorflow (Bryan Lim, et al. International Journal of Forecasting 2019)TabNet based on pytorch (Sercan O. Arik, et al. AAAI 2019)DoubleEnsemble based on LightGBM (Chuheng Zhang, et al. ICDM 2020)TCTS based on pytorch (Xueqing Wu, et al. ICML 2021)Transformer based on pytorch (Ashish Vaswani, et al. NeurIPS 2017)Localformer based on pytorch (Juyong Jiang, et al.)TRA based on pytorch (Hengxu, Dong, et al. KDD 2021)TCN based on pytorch (Shaojie Bai, et al. 2018)ADARNN based on pytorch (YunTao Du, et al. 2021)ADD based on pytorch (Hongshun Tang, et al.2020)IGMTF based on pytorch (Wentao Xu, et al.2021)HIST based on pytorch (Wentao Xu, et al.2021)KRNN based on pytorchSandwich based on pytorchYour PR of new Quant models is highly welcomed.The performance of each model on theAlpha158andAlpha360dataset can be foundhere.Run a single modelAll the models listed above are runnable withQlib. Users can find the config files we provide and some details about the model through thebenchmarksfolder. More information can be retrieved at the model files listed above.Qlibprovides three different ways to run a single model, users can pick the one that fits their cases best:Users can use the toolqrunmentioned above to run a model's workflow based from a config file.Users can create aworkflow_by_codepython script based on theonelisted in theexamplesfolder.Users can use the scriptrun_all_model.pylisted in theexamplesfolder to run a model. Here is an example of the specific shell command to be used:python run_all_model.py run --models=lightgbm, where the--modelsarguments can take any number of models listed above(the available models can be found  inbenchmarks). For more use cases, please refer to the file'sdocstrings.NOTE: Each baseline has different environment dependencies, please make sure that your python version aligns with the requirements(e.g. TFT only supports Python 3.6~3.7 due to the limitation oftensorflow==1.15.0)Run multiple modelsQlibalso provides a scriptrun_all_model.pywhich can run multiple models for several iterations. (Note: the script only supportLinuxfor now. Other OS will be supported in the future. Besides, it doesn't support parallel running the same model for multiple times as well, and this will be fixed in the future development too.)The script will create a unique virtual environment for each model, and delete the environments after training. Thus, only experiment results such asICandbacktestresults will be generated and stored.Here is an example of running all the models for 10 iterations:pythonrun_all_model.pyrun10It also provides the API to run specific models at once. For more use cases, please refer to the file'sdocstrings.Adapting to Market DynamicsDue to the non-stationary nature of the environment of the financial market, the data distribution may change in different periods, which makes the performance of models build on training data decays in the future test data.
So adapting the forecasting models/strategies to market dynamics is very important to the model/strategies' performance.Here is a list of solutions built onQlib.Rolling RetrainingDDG-DA on pytorch (Wendi, et al. AAAI 2022)Reinforcement Learning: modeling continuous decisionsQlib now supports reinforcement learning, a feature designed to model continuous investment decisions. This functionality assists investors in optimizing their trading strategies by learning from interactions with the environment to maximize some notion of cumulative reward.Here is a list of solutions built onQlibcategorized by scenarios.RL for order executionHereis the introduction of this scenario.  All the methods below are comparedhere.TWAPPPO: "An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization", IJCAL 2020OPDS: "Universal Trading for Order Execution with Oracle Policy Distillation", AAAI 2021Quant Dataset ZooDataset plays a very important role in Quant. Here is a list of the datasets built onQlib:DatasetUS MarketChina MarketAlpha360√√Alpha158√√Hereis a tutorial to build dataset withQlib.
Your PR to build new Quant dataset is highly welcomed.Learning FrameworkQlib is high customizable and a lot of its components are learnable.
The learnable components are instances ofForecast ModelandTrading Agent. They are learned based on theLearning Frameworklayer and then applied to multiple scenarios inWorkflowlayer.
The learning framework leverages theWorkflowlayer as well(e.g. sharingInformation Extractor, creating environments based onExecution Env).Based on learning paradigms, they can be categorized into reinforcement learning and supervised learning.For supervised learning, the detailed docs can be foundhere.For reinforcement learning, the detailed docs can be foundhere. Qlib's RL learning framework leveragesExecution EnvinWorkflowlayer to create environments.  It's worth noting thatNestedExecutoris supported as well. This empowers users to optimize different level of strategies/models/agents together (e.g. optimizing an order execution strategy for a specific portfolio management strategy).More About QlibIf you want to have a quick glance at the most frequently used components of qlib, you can try notebookshere.The detailed documents are organized indocs.Sphinxand the readthedocs theme is required to build the documentation in html formats.cddocs/
condainstallsphinxsphinx_rtd_theme-y# Otherwise, you can install them with pip# pip install sphinx sphinx_rtd_thememakehtmlYou can also view thelatest documentonline directly.Qlib is in active and continuing development. Our plan is in the roadmap, which is managed as agithub project.Offline Mode and Online ModeThe data server of Qlib can either deployed asOfflinemode orOnlinemode. The default mode is offline mode.UnderOfflinemode, the data will be deployed locally.UnderOnlinemode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too. The documents of the online mode can be found inQlib-Server. The online mode can be deployed automatically withAzure CLI based scripts. The source code of online data server can be found inQlib-Server repository.Performance of Qlib Data ServerThe performance of data processing is important to data-driven methods like AI technologies. As an AI-oriented platform, Qlib provides a solution for data storage and data processing. To demonstrate the performance of Qlib data server, we
compare it with several other data storage solutions.We evaluate the performance of several storage solutions by finishing the same task,
which creates a dataset (14 features/factors) from the basic OHLCV daily data of a stock market (800 stocks each day from 2007 to 2020). The task involves data queries and processing.HDF5MySQLMongoDBInfluxDBQlib -E -DQlib +E -DQlib +E +DTotal (1CPU) (seconds)184.4±3.7365.3±7.5253.6±6.7368.2±3.6147.0±8.847.6±1.07.4±0.3Total (64CPU) (seconds)8.8±0.64.2±0.2+(-)Eindicates with (out)ExpressionCache+(-)Dindicates with (out)DatasetCacheMost general-purpose databases take too much time to load data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions.
Such overheads greatly slow down the data loading process.
Qlib data are stored in a compact format, which is efficient to be combined into arrays for scientific computation.Related ReportsGuide To Qlib: Microsoft’s AI Investment Platform微软也搞AI量化平台？还是开源的！微矿Qlib：业内首个AI量化投资开源平台Contact UsIf you have any issues, please create issuehereor send messages ingitter.If you want to make contributions toQlib, pleasecreate pull requests.For other reasons, you are welcome to contact us by email(qlib@microsoft.com).We are recruiting new members(both FTEs and interns), your resumes are welcome!Join IM discussion groups:GitterContributingWe appreciate all contributions and thank all the contributors!Before we released Qlib as an open-source project on Github in Sep 2020, Qlib is an internal project in our group. Unfortunately, the internal commit history is not kept. A lot of members in our group have also contributed a lot to Qlib, which includes Ruihua Wang, Yinda Zhang, Haisu Yu, Shuyu Wang, Bochen Pang, andDong Zhou. Especially thanks toDong Zhoudue to his initial version of Qlib.GuidanceThis project welcomes contributions and suggestions.Here are somecode standards and development guidancefor submiting a pull request.Making contributions is not a hard thing. Solving an issue(maybe just answering a question raised inissues listorgitter), fixing/issuing a bug, improving the documents and even fixing a typo are important contributions to Qlib.For example, if you want to contribute to Qlib's document/code, you can follow the steps in the figure below.If you don't know how to start to contribute, you can refer to the following examples.TypeExamplesSolving issuesAnswer a question;issuingorfixinga bugDocsImprove docs quality;Fix a typoFeatureImplement arequested featurelikethis;Refactor interfacesDatasetAdd a datasetModelsImplement a new model,some instructions to contribute modelsGood first issuesare labelled to indicate that they are easy to start your contributions.You can find some impefect implementation in Qlib byrg 'TODO|FIXME' qlibIf you would like to become one of Qlib's maintainers to contribute more (e.g. help merge PR, triage issues), please contact us by email(qlib@microsoft.com).  We are glad to help to upgrade your permission.LicenceMost contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the right to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.