Package: pyluban
lubanWhy Need?The feature-processing module is used in different scenarios, such as: model training and model inference. Usually, we use Python to train models, and feature-processing is also done in Python. However, in model inference, the shorter the inference run time, the better. So we usually use C++/java/golang or other languages to do model inference. If feature-process module is written in many different languages, they may be inconsistencies. So, we decide to develop such a module in C++, which can be used by other different languages. In addition, we use a json configuration for feature processing, so that it will be easier to use.Supported libraries and toolsc/c++ lib: liblubanpython lib: pylubanConfiguration{"user_features":[{"name":"U1","expr":"A*B","type":1,"hash":false,"padding":0,"dim":1},{"name":"U2","expr":"C[int64]+D[int64]","type":0,"hash":false,"padding":0,"dim":1}],"item_features":[{"name":"I1","expr":"F[int64]+G[int64]","type":0,"hash":false,"padding":0,"dim":1},{"name":"I2","expr":"concat(H[string],\"TEST\")","type":0,"hash":true,"padding":0,"dim":1}],"groups":[["U1"],["U2"],["I1","I2"]]}featuresConfigure the relevant properties for each feature.PropertiesTypeEmptyDefaultDescriptionnamestringN-name of featureexprstringY""how to process featuretypeintN-0: int64,1: float3,2: string3: vector<int64>4: vector<float>5: vector<string>hashboolYfalsehash or notpaddingint64/float32Y0/0.0fill valuedimintY1dim of resultgroupsAfter processing, different features may be placed in different groups. Because the downstream user is pytorch, different types of data are inconvenient to put in a tensor. The final output order of the features is the same as the order configured in the group.All features configured in groups need to be configured in the features above.Parse ExpressionAfter installing the Python tool, theluban_parsertool is installed by default in /usr/local/bin. This tool is used to parse the above JSON configuration and generate the configuration using the JSON format used by C++. The expressions here are similar to Python's syntax. Originally, we wanted to use antlr to customize the DSL, but after thinking about it, we thought that it could be as simple as possible. As a result, it was finally decided to parse expressions using Python's built-in ast.Operatorsopr: +, -, *, /, %, **math function: round, floor, ceil, floorf, ceilf, log, exp, log10, log2, sqrt, abs, sin, sinh, asin,asinh, cos, cosh, acos, acosh, tan, tanh, atan, atanh, sigmoidaggravate function: min, max, variance, stddev, average, normtime function: year, month, day, hour, minute, second, date, now, date_diff, date_add, date_sub, from_unixtime, unix_timestampstring function: concat, substr, lower, upper, cross, reversetopk function: topki, topkf, topksother function: min_max, z_score, binarize, bucketize, box_cox, normalize// The following is a list of supported operations// If it is a template type, you can add the type feature when configuring the function name// e.g.: _add<int64_t>, _add<float>// If you do not add type information, the float type is defaulted// In practice, the tool does some built-in implicit data type conversionsT_add(T&a,T&b);T_sub(T&a,T&b);T_mul(T&a,T&b);T_div(T&a,T&b);int64_t_mod(int64_t&a,int64_t&b);float_pow(float&a,float&b);int64_t_round(float&x);int64_t_floor(float&x);int64_t_ceil(float&x);float_log(float&x);float_exp(float&x);float_log10(float&x);float_log2(float&x);float_sqrt(float&x);float_abs(float&x);float_sin(float&x);float_asin(float&x);float_sinh(float&x);float_asinh(float&x);float_cos(float&x);float_acos(float&x);float_cosh(float&x);float_acosh(float&x);float_tan(float&x);float_atan(float&x);float_tanh(float&x);float_atanh(float&x);float_sigmoid(float&x);Tmin(std::vector<float>&src);Tmax(std::vector<T>&src);floataverage(std::vector<T>&src);floatvariance(std::vector<T>&src);floatstddev(std::vector<T>&src);floatnorm(std::vector<T>&src,float&n);std::stringyear();std::stringmonth();std::stringday();std::stringdate();std::stringhour();std::stringminute();std::stringsecond();int64_tnow();std::stringfrom_unixtime(int64_t&ts,std::string&format);int64_tunix_timestamp(std::string&t,std::string&format);std::stringdate_add(std::string&s,int64_t&n);std::stringdate_sub(std::string&s,int64_t&n);int64_tdate_diff(std::string&d1,std::string&d2);std::stringreverse(std::string&s);std::stringupper(std::string&s);std::stringlower(std::string&s);std::stringsubstr(std::string&s,int64_t&start,int64_t&len);std::stringconcat(std::string&a,std::string&b);floatmin_max(T&v,T&min,T&max);floatz_score(float&v,float&mean,float&stdv);int64_tbinarize(T&v,T&threshold);int64_tbucketize(T&v,std::vector<T>&boundaries);floatbox_cox(float&v,float&lambda);std::vector<float>normalize(std::vector<T>&src,float&norm);std::vector<T>topk(std::vector<T>&src,int64_t&k);std::vector<std::string>cross(std::vector<std::string>&srcA,std::vector<std::string>&srcB);UsageFollow the installation prompts in the next section to compile and install the toolInclude header files and add libluban to dynamic link pathsSteps:step1: Configure the JSON fileuseluban_parserto process JSON configuration files and generate configuration files in JSON formatUse the configuration file in JSON format as a configuration input for C/C++/Golang/PythonInstallinstall On MacOS ARMpipinstallpylubanUnix Likepythonsetup.pyinstall--install-scripts=/usr/local/bin

pipinstallpylubanExamplesFeatures & FeaturesListThe structure of the features is as follows:// json format{"A":{// key is the name of feature"type":0,// type listed above"value":10// certain value},"B":{"type":1,"value":10.9},...}how to parse features in pythonimportjsonimportpylubanfeas={"A":{"type":0,"value":10},...}feas_str=json.dumps(feas)# arg is strfeatures=pyluban.Features(feas_str)# arg is str listfeatures=pyluban.Features([feas_str1,feas_str2,feas_str3])# arg is nilfeatures=pyluban.Features()# create a featureslistl=pyluban.FeatuersList()# add a valuel.append(features)print(l[0])print(l)l[0]=featuresFunctionThis is the configuration of function processing, usually,users generally do not use, will be used when doing unit tests, and are also explained here.{"func":"_add<int64_t>","name":"X","flag":3,"args":[],"vars":["A","B"]}PropertiesTypeDescriptionfuncstringfunction namenamestringname of generated featureflagint64Flag bits for each parameter: In the binary bit of flag, the ith bit is 1 to indicate that the input is a variable; Otherwise, it is constantargslist of dictsame structure of featuresvarslist of strvariable name listToolkitBy configuring the class for feature processing entry, the consumer mainly uses this class to process features.importpylubanimportluban_parser# this is the original config file"""{"features": [{"name": "A","expr": "(B+C)*F","type": 0,"hash": false,"padding": 0,"dim": 1},{"name": "B","type": 0,"padding": 0,"dim": 1},{"name": "C","type": 0,"padding": 0,"dim": 1},{"name": "D","expr": "concat(\"prefix-\", E)","type": 0,"hash": true,"padding": 0,"dim": 3}],"groups": [["A","B"],["D"]]}"""luban_parser.parser(input_file:str,output_file:str)# this is the output file of luban_parser"""{"transforms": [{"func": "_add<int64_t>","name": "anonymous_0","flag": 3,"args": [],"vars": ["B","C"]},{"func": "_mul<int64_t>","name": "A","flag": 3,"args": [],"vars": ["anonymous_0","F"]},{"func": "concat","name": "D","flag": 2,"args": [{"type": 2,"value": "prefix-"}],"vars": ["E"]}],"groups": [{"id": 0,"width": 2,"type": 0},{"id": 1,"width": 3,"type": 0}],"features": [{"name": "A","type": 0,"padding": 0,"group": 0,"offset": 0,"hash": false,"dim": 1},{"name": "B","type": 0,"padding": 0,"group": 0,"offset": 1,"hash": false,"dim": 1},{"name": "D","type": 0,"padding": 0,"group": 1,"offset": 0,"hash": true,"dim": 3}]}"""# config.json is the file after configuration processing,# and the above is an example of the file content of the configurationtoolkit=pyluban.Toolkit("config.json")RowsIn the case of only one feature, Rows is a list object with a length of the number of groups configured above, which can be accessed via subscripts and then converted into numpy structures.# r pyluban:Rows typer=toolkit.process(features:pyluban.Features)foriinrange(len(r)):print(np.asarray(r[i]))MatricesMatrices is a list object with a length of the number of groups configured above, which can be accessed by subscripts and then converted into a numpy structure.l=pyluban.FeaturesList()l.append(features:pyluban.Features)m=self.toolkit.process(l)foriinrange(len(m)):print(np.asarray(m[i]))