Package: factor-expr
Factor ExprFactor ExpressionHistorical DataFactor Values(LogReturn 30 :close)+2019-12-27~2020-01-14.pq=[0.01, 0.035, ...]Extreme fast factor expression & computation library for quantitative trading in Python.On a server with an E7-4830 CPU (16 cores, 2000MHz),
computing 48 factors over a dataset with 24.5M rows x 683 columns (12GB) takes 150s.Join[Discussions]for Q&A and feature proposal!FeaturesExpress factors inS-Expression.Compute factors in parallel over multiple factors and multiple datasets.UsageThere are three steps to use this library.Prepare the datasets into files. Currently, only theParquetformat is supported.Define factors usingS-Expression.Runreplayto compute the factors on the dataset.1. Prepare the datasetA dataset is a tabular format with float64 columns and arbitrary column names.
Each row in the dataset represents a tick, e.g. for a daily dataset, each row is one day.
For example, here is an OHLC candle dataset representing 2 ticks:df=pd.DataFrame({"open":[3.1,5.8],"high":[8.8,7.7],"low":[1.1,2.1],"close":[4.4,3.4]})You can use the following code to store the DataFrame into a Parquet file:df.to_parquet("data.pq")2. Define your factorsFactor Expruses the S-Expression to describe a factor.
For example, on a daily OHLC dataset, the 30 days log return on the columncloseis expressed as:fromfactor_exprimportFactorFactor("(LogReturn 30 :close)")Note, inFactor Expr, column names are referred by the:column-namesyntax.3. Compute the factors on the prepared datasetFollowing step 1 and 2, you can now compute the factors using thereplayfunction:fromfactor_exprimportFactor,replayresult=awaitreplay(["data.pq"],[Factor("(LogReturn 30 :close)")])The first parameter ofreplayis a list of dataset files and the second parameter is a list of Factors. This gives you the ability to compute multiple factors on multiple datasets.
Don't worry about the performance!Factor Exprallows you parallelize the computation over the factors as well as the datasets by settingn_factor_jobsandn_data_jobsin thereplayfunction.The returned result is a pandas DataFrame with factors as the column names andtimeas the index.
In case of multiple datasets are passed in, the results will be concatenated with the exact order of the datasets. This is useful if you have a scattered dataset. E.g. one file for each year.For example, the code above will give you a DataFrame looks similar to this:index(LogReturn 30 :close)00.23......Check out thedocstringofreplayfor more information!Installationpipinstallfactor-exprSupported FunctionsNotations:<const>means a constant, e.g.3.<expr>means either a constant or an S-Expression or a column name, e.g.3or(+ :close 3)or:open.Here's the full list of supported functions. If you didn't find one you need,
consider asking onDiscussionsor creating a PR!ArithmeticsAddition:(+ <expr> <expr>)Subtraction:(- <expr> <expr>)Multiplication:(* <expr> <expr>)Division:(/ <expr> <expr>)Power:(^ <const> <expr>)- compute<expr> ^ <const>Negation:(Neg <expr>)Signed Power:(SPow <const> <expr>)- computesign(<expr>) * abs(<expr>) ^ <const>Natural Logarithm after Absolute:(LogAbs <expr>)Sign:(Sign <expr>)Abs:(Abs <expr>)LogicsAny<expr>larger than 0 are treated astrue.If:(If <expr> <expr> <expr>)- if the first<expr>is larger than 0, return the second<expr>otherwise return the third<expr>And:(And <expr> <expr>)Or:(Or <expr> <expr>)Less Than:(< <expr> <expr>)Less Than or Equal:(<= <expr> <expr>)Great Than:(> <expr> <expr>)Greate Than or Equal:(>= <expr> <expr>)Equal:(== <expr> <expr>)Not:(! <expr>)Window FunctionsAll the window functions take a window size as the first argument. The computation will be done on the look-back window with the size given in<const>.Sum of the window elements:(Sum <const> <expr>)Mean of the window elements:(Mean <const> <expr>)Min of the window elements:(Min <const> <expr>)Max of the window elements:(Max <const> <expr>)The index of the min of the window elements:(ArgMin <const> <expr>)The index of the max of the window elements:(ArgMax <const> <expr>)Stdev of the window elements:(Std <const> <expr>)Skew of the window elements:(Skew <const> <expr>)The rank (ascending) of the current element in the window:(Rank <const> <expr>)The value<const>ticks back:(Delay <const> <expr>)The log return of the value<const>ticks back to current value:(LogReturn <const> <expr>)Rolling correlation between two series:(Correlation <const> <expr> <expr>)Rolling quantile of a series:(Quantile <const> <const> <expr>), e.g.(Quantile 100 0.5 <expr>)computes the median of a window sized 100.Warm-up Period for Window FunctionsFactors containing window functions require a warm-up period. For example, for(Sum 10 :close), it will not generate data until the 10th tick is replayed.
In this case,replaywill writeNaNinto the result during the warm-up period, until the factor starts to produce data.
This ensures the length of the factor output will be as same as the length of the input dataset. You can use thetrimparameter to let replay trim off the warm-up period before it returns.Factors Failed to ComputeFactor Exprguarantees that there will not be anyinf,-inforNaNappear in the result, except for the warm-up period. However, sometimes a factor can fail due to numerical issues. For example,(Pow 3 (Pow 3 (Pow 3 :volume)))might overflow and becomeinf, and1 / infwill becomeNaN.Factor Exprwill detect these situations and mark these factors as failed. The failed factors will still be returned in the replay result, but the values in that column will be allNaN. You can easily remove these failed factors from the result by usingpd.DataFrame.dropna(axis=1, how="all").I Want to Have a Time Index for the ResultThereplayfunction optionally accepts aindex_colparameter.
If you want to set a column from the dataset as the index of the returned result, you can do the following:fromfactor_exprimportFactor,replaypd.DataFrame({"time":[datetime(2021,4,23),datetime(2021,4,24)],"open":[3.1,5.8],"high":[8.8,7.7],"low":[1.1,2.1],"close":[4.4,3.4],}).to_parquet("data.pq")result=awaitreplay(["data.pq"],[Factor("(LogReturn 30 :close)")],index_col="time",)Note, accessing thetimecolumn from factor expressions will cause an error.
Factor expressions can only readfloat64columns.APIThere are two components inFactor Expr, aFactorclass and areplayfunction.FactorThe factor class takes an S-Expression to construct. It has the following signature:classFactor:def__init__(sexpr:str)->None:"""Construct a Factor using an S-Expression"""defready_offset(self)->int:"""Returns the first index after the warm-up period.For non-window functions, this will always return 0."""def__len__(self)->int:"""Returns how many subtrees contained in this factor tree.Example-------`(+ (/ :close :open) :high)` has 5 subtrees, namely:1. (+ (/ :close :open) :high)2. (/ :close :open)3. :close4. :open5. :high"""def__getitem__(self,i:int)->Factor:"""Get the i-th subtree of the sequence from the pre-order traversal of the factor tree.Example-------`(+ (/ :close :open) :high)` is traversed as:0. (+ (/ :close :open) :high)1. (/ :close :open)2. :close3. :open4. :highConsequently, f[2] will give you `Factor(":close")`."""defdepth(self)->int:"""How deep is this factor tree.Example-------`(+ (/ :close :open) :high)` has a depth of 2, namely:1. (+ (/ :close :open) :high)2. (/ :close :open)"""defchild_indices(self)->List[int]:"""The indices for the children of this factor tree.Example-------The child_indices result of `(+ (/ :close :open) :high)` is [1, 4]"""defreplace(self,i:int,other:Factor)->Factor:"""Replace the i-th node with another subtree.Example-------`Factor("+ (/ :close :open) :high").replace(4, Factor("(- :high :low)")) == Factor("+ (/ :close :open) (- :high :low)")`"""defcolumns(self)->List[str]:"""Return all the columns that are used by this factor.Example-------`(+ (/ :close :open) :high)` uses [:close, :open, :high]."""defclone(self)->Factor:"""Create a copy of itself."""replayReplay has the following signature:asyncdefreplay(files:Iterable[str],factors:List[Factor],*,reset:bool=True,batch_size:int=40960,n_data_jobs:int=1,n_factor_jobs:int=1,pbar:bool=True,verbose:bool=False,output:Literal["pandas","pyarrow","raw"]="pandas",)->Union[pd.DataFrame,pa.Table]:"""Replay a list of factors on a bunch of data.Parameters----------files: Iterable[str | pa.Table]Paths to the datasets. Or already read pyarrow Tables.factors: List[Factor]A list of Factors to replay.reset: bool = TrueWhether to reset the factors. Factors carries memory about the data they already replayed. If you are callingreplay multiple times and the factors should not starting from fresh, set this to False.batch_size: int = 40960How many rows to replay at one time. Default is 40960 rows.n_data_jobs: int = 1How many datasets to run in parallel. Note that the factor level parallelism is controlled by n_factor_jobs.n_factor_jobs: int = 1How many factors to run in parallel for **each** dataset.e.g. if `n_data_jobs=3` and `n_factor_jobs=5`, you will have 3 * 5 threads running concurrently.pbar: bool = TrueWhether to show the progress bar using tqdm.verbose: bool = FalseIf True, failed factors will be printed out in stderr.output: Literal["pyarrow" | "raw"] = "pyarrow"The return format, can be pyarrow Table ("pyarrow") or un-concatenated pyarrow Tables ("raw")."""