Package: evaluation-framework
This repository contains a (software)evaluation frameworkto perform evaluation and comparison onnode embedding techniques. It can be easily extended by also considering edges into the evaluation. The provided tasks range from Machine Learning (ML) (classification, regression, and clustering) and semantic tasks (entity relatedness and document similarity) tosemantic analogies. The framework is designed to be extended with additional tasks. It is  useful both for embedding algorithm developers and users. On one side, when a new embedding algorithm is defined, there is the need to evaluate it upon tasks it was created for. On the other side, users can be interested in performing particular tests and choosing the embedding algorithm that performs best for their application. Our goal is to address both situations providing a ready-to-use framework that can be customized and easily extended. A preliminary overview of the framework and its results can be accessed throughA Configurable Evaluation Framework for Node Embedding Techniques.We provide the framework as a command-line tool and we are working to the REST API development (you can inspect thedevbranch).Please referhttps://mariaangelapellegrino.github.io/Evaluation-Framework/to access the complete readme as a webpage.Framework structure and extension pointsIt is a diagrammatic representation of the involved actors in the framework and their interactions. The blue boxes represent abstract classes, while the white boxes represent concrete classes. If A<<extends>>B, A is the concrete class that extends and makes the abstract behavior of A concrete. If A<<instantiates>>B, A creates an instance of B.<<uses>>B, A is dependent on B.The starting point of the evaluation is theEvaluation Managerwhich is the orchestrator of the whole evaluation and it is in charge ofverify the correctness of the parameters set by the user,instantiating the correct data manager according to the data format provided by the user,determining which task(s) the user asked for,managing the storage of the results.Depending on the file format, the correspondingdata managerdecides how to read the vector file content and how to manage the access to it (e.g., if the whole content has to be load in memory). Eachdata managerhas tomanage the reading of the gold standard datasets,manage the reading of the input file,determine how to merge each gold standard dataset and the input file.The behavior of the data manager is modeled by theabstract data manager, implemented by a concrete data manager based on the input file format and it refined by a task data manager.Each task is modeled as a pair oftask managerandmodel.
Thetask manageris in charge ofmerging the input file and each gold standard file (if more than one is provided) (by exploiting the data manager),instantiating and training a model for each configuration to test,collecting and storing results computed by the model.Each task can decide if the missing entities (i.e., the entities required into the gold standard file, but absent into the input file) will affect the final result of the task or not.To extend the evaluation also to edges, it is enough to create a gold-standard dataset containing edges and related ground truth.TasksThe implemented tasks are:Machine LearningClassificationRegressionClusteringSemantic tasksEntity RelatednessDocument SimilaritySemantic AnalogiesEach task follows the same workflow:the task manager asks data manager to merge each gold standard dataset and the input file and keeps track of both the retrieved vectors and themissing entities,  i.e.,  entities  required  by  the  gold  standard  dataset,  but  absent  inthe input file;a model for each configuration is instantiated and trained;the missing entities are managed: it is up to the task to decide if they should affect the final result or they can be simply ignored;the scores are calculated and stored.We will separately analyse each task, by detailing the gold standard datasets, the configuration of the model(s), and the computed evaluation metrics.Framework detailsParametersThe parameters of the evaluation are:vectors_file: vector file path,mandatoryvector_file_format: {TXT, HDF5},default TXTvectors_size: numeric value,default 200tasks: list ofClassification,Regression,Clustering,EntityRelatedness,SemanticAnalogiesandDocumentSimilarity,default _allparallel: boolean,default Falsedebugging_mode: boolean,default False|           -similarity_metric:Sklearn affinity metrics,default Cosineanalogy_function: None (to use the _default_analogy_function_), otherwise a function handler. The semantic analogy tasks takes a quadruplet of vectors (a,b,c,d) and it verifies if by manipulating the first three vectors it is possible to predict the last one. The manipulation happens by the analogy function.def default_analogy_function(a,b,c){return b-a+c}Any other handler to function has to take as input 3 vectors and return a predicted vector, which will be compared with the last vector in each semantic quadruplet.top_knumeric value,default 2. It is used by the semantic analogy task: the predicted vector is compared with the top_k closest vectors. If the actual _d_ vector is among those, the task is considered correct.compare_with: list of run IDs,default _allVector file formatThe input file can be provided either as a plain text (also calledTXT) file or as anHDF5.TheTXTfile must be a white-space separated value file with a line for each embedded entity. Each row must contain the IRI of the embedded entity - without angular brackets - and its vector representation.TheHDF5vectors file must be an H5 file with a singlegroupcalledVectors.
In this group, there must be adatasetfor each entity with thebase32 encodingof the entity named as the dataset name and the embedded vector as its value.Running detailsUsers can customize the evaluation settings by:specifying parameters on the command line (useful when only a few settings must be specified and the user desires to use the default value for most of the parameters);organizing them in an XML file (especially useful when there is the need to define most of the parameters);passing them to a function that starts the evaluation.In theexamplefolder of the project on GitHub, there are examples for the different ways to provide the parameters.To execute one of them you can move the desiredmainfile at the top level of the project and then run it.Note: The tasks can be executed sequentially or in parallel. If the code raises MemoryError it means that the tasks need more memory than the one available. In that case, run all the tasks sequentially.Results storageFor each task and for each file used as a gold standard, the framework will createan output file that contains a reference to the file used as a gold standard and all the information related to evaluation metric(s) provided by each task,a file containing all themissingentities,a log file reporting extra information, occurred problems, and execution time,information related to the comparison with previous runs.In particular, about the comparison, it reports the values effectively considered in the comparison and the ranking of the current run upon the other ones. The results of each run are stored in the directoryresults/result_<starting time of the execution>generated by the evaluation manager in the local path.