Package: comex
Tree Sitter Multi Codeview GeneratorTree Sitter Multi Codeview Generator aims to generate combined multi-code view graphs that can be used with various types of machine learning models (sequence models, graph neural networks, etc).Comexcomexis a rebuild of Tree Sitter Multi Codeview Generator for easier invocation as a Python package. This rebuild also includes a cli interface. Currently,comexgenerates codeviews for Java and C#, for both method-level and file-level code snippets.comexcan be used to generate over $15$ possible combinations of codeviews for both languages (complete listhere).comexis designed to be easily extendable to various programming languages. This is primarliy because we usetree-sitterfor parsing, a highly efficient incremental parser that supports over $40$ languages. If you wish to add support for more languages, please refer to thecontributingguide.If you wish to learn more about the approach taken, here are some conference talks and publications:ASE 2023 demonstration:COMEX: A Tool for Generating Customized Source Code RepresentationsICSE 2023 tutorial:The Landscape of Source Code Representation Learning in AI-Driven Software Engineering TasksCite ComexIf you use Comex in your research, please cite our work by using the following BibTeX entry:@misc{das2023comex,
      title={COMEX: A Tool for Generating Customized Source Code Representations}, 
      author={Debeshee Das and Noble Saji Mathews and Alex Mathai and Srikanth Tamilselvam and Kranthi Sedamaki and Sridhar Chimalakonda and Atul Kumar},
      year={2023},
      eprint={2307.04693},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}Installation from PyPicomexis published on the Python Registry and can be easily installed via pip:pip install comexNote: You would need to install GraphViz(dot) so that the graph visualizations are generatedInstallation from sourceTo setupcomexfor development using the source code in your python environment:pip install -r requirements-dev.txtThis performs an editable install, meaning that comex would be available throughout your environment (particularly relevant if you use conda or something of the sort). This means now you can interact and import fromcomexjust like any other package while remaining standalone but also reflecting any code side updates without any other manual stepsUsage as a CLIThis is the recommended way to get started withcomexas it is the most user friendlyThe attributes and options supported by the CLI are well documented and can be viewed by running:comex --helpFor example, to generate a combined CFG and DFG graph for a java file, you can run:comex --lang "java" --code-file ./test.java --graphs "cfg,dfg"Usage as a Python PackageThe comex package can be used by importing required drivers as follows:fromcomex.codeviews.combined_graph.combined_driverimportCombinedDriverCombinedDriver(src_language=lang,src_code=code,output_file="output.json",graph_format=output,codeviews=codeviews)In most cases the required combination can be obtained via thecombined_drivermodule as shown above.src_language: denotes one of the supported languaged hence currently "java" or "cs"

src_code: denotes the source code to be parsed

output_file: denotes the output file to which the generated graph is written

graph_format: denotes the format of the output graph. Currently supported formats are "dot" and "json". To generate both pass "all"

codeviews: refers to the configuration passed for each codeviewLimitationsWhilecomexprovidesmethod-levelandfile-levelsupport for both Java and C#, it's important to note the following limitations and known issues:JavaNo Inter-file Analysis Support: The tool currently does not support codeviews that involve interactions between multiple Java files. It is designed to generate codeviews for individual Java files only.Syntax Errors in Code: Despite supporting non-compileable code, to ensure accurate codeviews, the input Java code must be free of syntax errors. Code with syntax errors may not be correctly parsed and displayed in the generated codeviews.Limited Support for Function Call Arguments: The tool does not provide proper support for when a function call is passed as an argument to another function call in Java code. The resulting codeview might not accurately represent the intended behavior in such cases.C#In addition to the limitations mentioned for Java, the tool has the following limitations specific to C#:No Support for Lambda Functions and Arrow Expressions: The tool does not support codeviews involving lambda functions and arrow expressions in C#. The generated codeviews may not accurately represent these language features.No Support for Compiler Directives: Compiler directives, such as pragma directives, are not supported by the tool. Code involving such directives may not be properly displayed in the generated codeviews.Incomplete Operator Declaration Support: The tool may have limited support for operator declarations in C#. Certain constraints and edge cases related to operator overloading might not be fully captured in the generated codeviews.Limited Support for Inheritance and Abstraction: The tool's support for inheritance and abstraction in C# is not fully comprehensive. Codeviews involving complex inheritance hierarchies or advanced abstraction patterns may not be accurately represented.Please note that while we continuously work to improve the tool and address these limitations, the current implementation may not be perfect. We appreciate your understanding and encourage you to provide feedback and report any issues you encounter, as this helps us enhance the tool's capabilities.Output Examples:Combined simple AST+CFG+DFG for a simple Java program that finds the maximum among 2 numbers:Below we present more examples of input code snippets and generated codeviews for both Java and C#.CLI Command:comex--lang"java"--code-filesample/example.java--graphs"cfg,dfg"Java Code Snippet:importjava.io.ByteArrayInputStream;importjava.io.InputStream;importjava.util.ArrayList;importjava.util.List;importjava.util.Scanner;publicclassMain{//    static String INPUT = "5\n3 2 2 4 1\n1 2 2 2 1";staticStringINPUT="";publicstaticvoidmain(String[]args){InputStreamis=INPUT.isEmpty()?System.in:newByteArrayInputStream(INPUT.getBytes());Scannerscanner=newScanner(is);finalintn=scanner.nextInt();List<Position>positionList=newArrayList<>(n);for(inti=0;i<n;i++){positionList.add(newPosition(scanner.nextInt(),scanner.nextInt(),scanner.nextInt()));}System.out.println(solve(positionList)?"Yes":"No");}staticclassPosition{intt;intx;inty;publicPosition(intt,intx,inty){this.t=t;this.x=x;this.y=y;}}staticbooleansolve(List<Position>positionList){PositioncurrentPosition=newPosition(0,0,0);for(inti=0;i<positionList.size();i++){PositionnextPosition=positionList.get(i);if(!possibleMove(currentPosition.t,nextPosition.t,currentPosition.x,nextPosition.x,currentPosition.y,nextPosition.y)){returnfalse;}currentPosition=nextPosition;}returntrue;}staticbooleanpossibleMove(intt1,intt2,intx1,intx2,inty1,inty2){inttDiff=t2-t1;intabsX=Math.abs(x1-x2);intabsY=Math.abs(y1-y2);if(absX+absY<=tDiff){if(tDiff%2==(absX+absY)%2){returntrue;}}returnfalse;}}Generated Codeview:CLI Command:comex--lang"cs"--code-filesample/example.cs--graphs"cfg,dfg"C# Code Snippet:publicclassDFG_A2{publicvoidmain(string[]args){BufferedReaderbr=newBufferedReader(newInputStreamReader(System.in));// {}Stringstr=br.attribute1;// {3}str=br.attribute2.method1();// {3}br.attribute1=br.attribute2;// {3,5}br.method2(br.attribute1,br.attribute2);// {3,5,6}BufferedReaderbr2=br;// {5,6,7}br.method3();// {5,6,7}intj=br2.attribute1.method2(3,4);// {8}}}Generated Codeview:More examples and results can be found in thetests/datadirectoryCode OrganizationThe code is structured in the following way:For each code-view, first the source code is parsed using the tree-sitter parser and then the various code-views are generated. In thetree_parserdirectory, the Parser and ParserDriver is implemented with various funcitonalities commonly required by all code-views. Language-specific features are further developed in the language-specific parsers also placed in this directory.Thecodeviewsdirectory contains the core logic for the various codeviews. Each codeview has a driver class and a codeview class, which is further inherited and extended by language in case of code-views that require language-specific implementation.Thecli.pyfile is the CLI implementation. The drivers can also be directly imported and used like a python package. It is responsible for parsing the source code and generating the codeviews.TestingThe repo is setup to automatically perform CI tests on making pulls to main and development branches.
To test locally:Run specific testSay you wish to runtest_cfgfunctionDrop the'[...]'part to run all tests in a fileformatted as [extension-filename]no-cov prevents coverage report from being printedpytest -k 'test_cfg[cs-test7]' --no-covRun all tests and get coverage reportpytestAnalyze the deviation report given bydeepdiffby using the verbose output.
This will help quickly figure out difference from the gold filepytest -k 'test_cfg[cs-test7]' --no-cov -vvPublishingMake sure to bump the version insetup.cfg.Then run the following commands:rm-rfbuilddist
pythonsetup.pysdistbdist_wheelThen upload it to PyPI usingtwine(pip install twineif not installed):twineuploaddist/*About the IBM OSCP ProjectThis tool was developed for research purposes as a part of the OSCP Project. Efficient representation of source code is essential for various software engineering tasks using AI pipelines such as code translation, code search and code clone detection. Code Representation aims at extracting the both syntactic and semantic features of source code and representing them by a vector which can be readily used for the downstream tasks. Multiple works exist that attempt to encode the code as sequential data to easily leverage state of art NN models like transformers. But it leads to a loss of information. Graphs are a natural representation for the code but very few works(MVG-AAAI’22) have tried to represent the different code features obtained from different code views like Program Dependency Graph, Data Flow Graph etc. as a multi-view graph. In this work, we want to explore more code views and its relevance to different code tasks as well as leverage transformers model for the multi-code view graphs. We believe such a work will help toEstablish influence of specific code views for common tasksDemonstrate how graphs can combined with transformersCreate re-usable modelsTeamThis tool is based on the ongoing joint research effort between IBM andRisha LabatIIT Tirupatito explore the effects of different code representations on code based tasks involving:Srikanth TamilselvamSridhar ChimalakondaAlex MathaiDebeshee DasNoble Saji MathewsKranthi Sedamaki