Package: hityper
HiTyperThis is the tool released in the ICSE 2022 paper"Static Inference Meets Deep Learning: A Hybrid Type InferenceApproach for Python".Updates8 Aug, 2022:We add a new commandhityper preprocessto transform the json files in ManyTypes4py datasets into thegroundtruth.jsonanddetailed_groundtruth.jsonfiles HiTyper needs underhityper eval.We also add a new option-ginhityper findusertypeto collect theusertypes.jsonHiTyper needs underhityper evalaccording to the groundtruth filegroundtruth.json.WorkflowHiTyper is a hybrid type inference tool built upon Type Dependency Graph (TDG), the typical workflow of it is as follows:For more details, please refer to thepaper.MethdologyThe general methdology of HiTyper is:Static inference is accurate but suffer from coverage problem due to dynamic featuresDeep learning models are feature-agnostic but they can hardly maintain the type correctness and are unable to predict unseen user-defined typesThe combination of static inference and deep learning shall complement each other and improve the coverage while maintaining the accuracy.InstallInstall HiTyper from sourceTo use HiTyper on your own computer, you can build from source: (If you need to modify the source code of HiTyper, please use this method and re-run thepip install .after modification each time)gitclonehttps://github.com/JohnnyPeng18/HiTyper.git&&cdHiTyper
pipinstall.Install HiTyper usingpipYou can install the latest version of HiTyper by using the following command:pipinstallhityperRequirements:Python>=3.9LinuxHiTyper requires running under Python >= 3.9 because there are a lot of new nodes introduced on AST from Python 3.9. However, HiTyper can analyze most files written under Python 3 since Python's AST is backward compatible.You are recommended to useAnacondato create a clean Python 3.9 environment and avoid most dependency conflicts:condacreate-nhityperpython=3.9UsageCurrently HiTyper has the following command line options: (Some important settings are stored in fileconfig.py, you may need to modify it before running HiTyper)findusertypeusage:hityperfindusertype[-h][-sSOURCE][-pREPO][-gGROUNDTRUTH][-cCORE][-v][-dOUTPUT_DIRECTORY]optionalarguments:-h,--helpshowthishelpmessageandexit-sSOURCE,--sourceSOURCEPathtoaPythonsourcefile-pREPO,--repoREPOPathtoaPythonproject-gGROUNDTRUTH,--groundtruthGROUNDTRUTHPathtoagroundtruthfile-cCORE,--coreCORENumberofcorestousewhencollectinguser-definedtypes-v,--validateValidatetheimporteduser-definedtypesbyfindingtheirimplementations-dOUTPUT_DIRECTORY,--output_directoryOUTPUT_DIRECTORYPathtothestoretheusertypesExample of collecting user-defined types in source files:hityperfindusertype-spython_project_repo/test.py-ppython_project_repo-v-doutputsThis command generates the user-defined types collected by HiTyper and save them as.jsonfiles underoutputs/folder.-poption is required here, if you do not specify-s, the HiTyper will collect user-defined types in all files of repo specified by-p.[Newly Added 6 Aug]We add a option to automatically generate all user-defined type files that a ground truth dataset needs to evaluate HiTyper.Example of collecting user-defined types in groundtruth datasets:hityperfindusertype-ggroundtruth.json-prepo_prefix-c60-doutputsThis command generates the user-defined types in files indicates bygroundtruth.jsoncollected by HiTyper and save them as.jsonfiles underoutputs/folder.For thegroundtruth.json, you need to use the same file inhityper evalcommand or generate it by usinghityper preprocesscommand.-p repo_prefixis an optional argument here, if the filenames ingroundtruth.jsonare the absolute paths then you do not need to specify-p, otherwise use-pto indicate which folder the source files are stored.The collection of all user-defined types for a large dataset is quite slow, try to specify a large number of cores used to make this process faster.gentdghitypergentdg[-h][-sSOURCE]-pREPO[-o][-lLOCATION][-a][-c][-dOUTPUT_DIRECTORY][-f{json,pdf}]optionalarguments:-h,--helpshowthishelpmessageandexit-sSOURCE,--sourceSOURCEPathtoaPythonsourcefile-pREPO,--repoREPOPathtoaPythonproject-o,--optimizeRemoveredundantnodesinTDG-lLOCATION,--locationLOCATIONGenerateTDGforaspecificfunction-a,--alias_analysisGeneratealiasgraphsalongwithTDG-c,--call_analysisGeneratecallgraphsalongwithTDG-dOUTPUT_DIRECTORY,--output_directoryOUTPUT_DIRECTORYPathtothegeneratedTDGs-f{json,pdf},--output_format{json,pdf}FormatsofoutputTDGsExample:hityper gentdg -s python_project_repo/test.py -p python_project_repo -d outputs -f json -oThis command generates the TDG for all functions in filepython_project_repo/test.pyand save them intooutputsfolder.Note that if you choosejsonformat to save TDG, it will be only ONEjsonfile that contains all TDGs in the source file. However, if you choosepdfformat to save TDG, then there will be multiplepdffiles and each one correspond to one function in the source file. This is because a pdf file can hardly contain a large TDG for every functions.For the location indicated by-l, use the formatfuncname@classnameand useglobalas the classname if the function is a global function.HiTyper usesPyCGto build call graphs in call analysis. Alias analysis and call analysis are temporarily built-in but HiTyper does not use them in inference. Further updates about them will be involved in HiTyper.inferhityperinfer[-h][-sSOURCE]-pREPO[-lLOCATION][-dOUTPUT_DIRECTORY][-mRECOMMENDATIONS][-t][-nTOPN]optionalarguments:-h,--helpshowthishelpmessageandexit-sSOURCE,--sourceSOURCEPathtoaPythonsourcefile-pREPO,--repoREPOPathtoaPythonproject-lLOCATION,--locationLOCATIONTypeinferenceforaspecificfunction-dOUTPUT_DIRECTORY,--output_directoryOUTPUT_DIRECTORYPathtothegeneratedTDGs-mRECOMMENDATIONS,--recommendationsRECOMMENDATIONSPathtotherecommendationsgeneratedbyaDLmodel-t,--type4pyUseType4Pyastherecommendationmodel-nTOPN,--topnTOPNIndicatethetopnpredictionsfromDLmodelsusedbyHiTyperExample:hityper infer -s python_project_repo/test.py -p python_project_repo -d outputs -n 1 -tThis command generates the inferred types for all variables, arguments and return values in the source file and save them intooutputfolder.If you do not specify-mor-toption, then HiTyper will only use the static inference part to infer types. Static inference generally takes several minutes.For the location indicated by-l, use the formatfuncname@classnameand useglobalas the classname if the function is a global function.Recommendation Model:Note that HiTyper natively supports the recommendations from Type4Py and it invokes the following API provided by Type4Py to get recommendations if you use option-t:https://type4py.com/api/predict?tc=0This will upload your file to the Type4Py server!If you do not want to upload your file, you can use thedockerprovided by Type4Py and changes the API inconfig.pyinto:http://localhost:PORT/api/predict?tc=0According to our experiments, the Type4Py model has much lower performance by quering the API above, you are suggested to train the model locally and generate the recommendation file which can be passed to-m.Note: HiTyper's performance deeply depends on the maximum performance of recommendation model (especially the performance to predict argument types). Type inference of HiTyper can fail if the recommendation model cannot give a valid prediction while static inference does not work!If you want to use another more powerful model, you write code like__main__.pyto adapt HiTyper to your DL model.evalhitypereval[-h]-gGROUNDTRUTH-cCLASSIFIED_GROUNDTRUTH-uUSERTYPE[-mRECOMMENDATIONS][-t][-nTOPN]optionalarguments:-h,--helpshowthishelpmessageandexit-gGROUNDTRUTH,--groundtruthGROUNDTRUTHPathtoagroundtruthdataset-cCLASSIFIED_GROUNDTRUTH,--classified_groundtruthCLASSIFIED_GROUNDTRUTHPathtoaclassifiedgroundtruthdataset-uUSERTYPE,--usertypeUSERTYPEPathtoapreviouslycollecteduser-definedtypeset-mRECOMMENDATIONS,--recommendationsRECOMMENDATIONSPathtotherecommendationsgeneratedbyaDLmodel-t,--type4pyUseType4Pyastherecommendationmodel-nTOPN,--topnTOPNIndicatethetopnpredictionsfromDLmodelsusedbyHiTyperExample:hitypereval-ggroundtruth.json-cdetailed_groundtruth.json-uusertypes.json-n1-tThis command evaluates the performance of HiTyper on a pre-defined groundtruth dataset. It will output similar results like stated inExperiment Resultspart.Before evaluating Hityper using this command, please usehityper findusertypecommand to generateusertypes.json. This typically takes several hours, depending on the number of files.This option is designed only for future research evaluation.Preprocessusage:hityperpreprocess[-h]-pJSON_REPO[-dOUTPUT_DIRECTORY]optionalarguments:-h,--helpshowthishelpmessageandexit-pJSON_REPO,--json_repoJSON_REPOPathtotherepoofJSONfiles-dOUTPUT_DIRECTORY,--output_directoryOUTPUT_DIRECTORYPathtothetransformeddatasetsExample:hityperpreprocess-pManyTypes4PyDataset/processed_projects_complete-doutputsThis command transforms the json files in ManyTypes4Py datasets into thegroundtruth.jsonanddetailed_groundtruth.jsonfiles that required by thehityper evalcommand.This command is to facilitate the researchers that use ManyTypes4Py dataset and want to evaluate HiTyper in it.If you want to run HiTyper in other datasets, please follow the same logic intransformDatasetfunction ofHiTyper/hityper/utils.pyto write a script.Experiment ResultsDataset:The following results are evaluated using theManyTypes4Pydataset.Since the original dataset does not contain Python source files, to facilitate future research, we here also attached alinkfor the Python source files HiTyper uses to infer types. Attached dataset is not identical with the original one because the original one contains some GitHub repos that do not allow open access or have been deleted.Note that as stated in the paper, there exists few cases (such as subtypes and same types with different names) that HiTyper should be correct but still counted as wrong in the evaluation process.Metrics:For the definition of metrics used here, please also refer to the paper. These metrics can be regarded as a kind of "recall", which evaluates the coverage of HiTyper on a specific dataset. We do not show the "precision" here as HiTyper only outputs results when it does not observe any violations with current typing rules and TDG.Only using the static inference part:CategoryExact MatchMatch to ParametricPartial MatchSimple Types59.00%59.47%62.15%Generic Types55.50%69.68%71.90%User-defined Types40.40%40.40%44.30%Arguments7.65%8.05%14.39%Return Values58.71%64.61%69.06%Local Variables61.56%65.66%67.05%You can use the following command to reproduce the above results:hitypereval-gManyTypes4Py_gts_test_verified.json-cManyTypes4Py_gts_test_verified_detailed.json-uManyTypes4Py_test_usertypes.jsonWe do not show the performance of HiTyper integrating different DL models here since there are many factors impacting the performance of DL models such as datasets, hyper-parameters, etc. Please align the performance by yourself before utilizing recommendations from DL models.What's more, we are currently working on building a DL model that's more suitable for HiTyper. Stay tuned!Other datasets:If you want to evaluate HiTyper on other datasets, please generate files with the same format withManyTypes4Py_gts_test_verified.json,ManyTypes4Py_gts_test_verified_detailed.json, or you can modify the code in__main__.py. To check a type's category, you can usehityper.typeobject.TypeObject.checkType().In any case, you must also prepare the source files for static analysis.Old results:If you want the exact experiment results stated in the paper, please download them at thislink.TodoAdd supports for inter-procedural analysisAdd supports for types from third-party modulesAdd supports for external function callsAdd supports for stub filesCite UsIf you use HiTyper in your research, please cite us:@inproceedings{peng22hityper,
author ={Peng, Yun and Gao, Cuiyun and Li, Zongjie and Gao, Bowei and Lo, David and Zhang, Qirun and Lyu, Michael},
title ={Static Inference Meets Deep Learning: A Hybrid Type Inference Approach for Python},
year ={2022},
isbn ={9781450392211},
publisher ={Association for Computing Machinery},
address ={New York, NY, USA},
url ={https://doi.org/10.1145/3510003.3510038},
doi ={10.1145/3510003.3510038},
booktitle ={Proceedings of the 44th International Conference on Software Engineering},
pages ={2019–2030},
numpages ={12},
location ={Pittsburgh, Pennsylvania},
series ={ICSE '22}}ContactWe actively maintain this project and welcome contributions.If you have any question, please contactresearch@yunpeng.work.