Package: evalne
EvalNE: A Python library for evaluating Network Embedding methodsRead The Docs »Table of ContentsAbout EvalNEFor methodologistsFor practitionersInstallationUsageAs a command line toolAs an APIOutputContributingLicenseCitationAbout EvalNEThis repository provides the source code for EvalNE, an open-source Python
library designed for assessing and comparing the performance of Network
Embedding (NE) methods on Link Prediction (LP), Sign prediction (SP),
Network Reconstruction (NR) and Node Classification (NC) tasks.
The library intends to simplify these complex and time consuming evaluation
processes by providing automation and abstraction of tasks such as
hyper-parameter tuning and model validation, node and edge sampling, node-pair
embedding computation, results reporting and data visualization.The library can be used both as a command line tool and an API. In its current
version, EvalNE can evaluate unweighted directed and undirected simple networks.The library is maintained by Alexandru Mara (alexandru.mara(at)ugent.be). The full
documentation of EvalNE is hosted byRead the Docsand can be foundhere.For MethodologistsA command line interface in combination with a configuration file (describing datasets,
methods and evaluation setup) allows the user to evaluate any embedding method and compare
it to the state of the art or replicate the experimental setup of existing papers without
the need to write additional code. EvalNE does not provide implementations of any NE methods
but offers the necessary environment to evaluate any off-the-shelf algorithm.
Implementations of NE methods can be obtained from libraries
such asOpenNEorGEMas well as directly from the web pages of the authors e.g.Deepwalk,Node2vec,LINE,PRUNE,Metapath2vec,CNE.EvalNE does, however, includes the following LP heuristics for both directed and
undirected networks (in and out node neighbourhoods), which can be used as
baselines for different downstream tasks:Random PredictionCommon NeighboursJaccard CoefficientAdamic Adar IndexPreferential AttachmentResource Allocation IndexCosine SimilarityLeicht-Holme-Newman indexTopological OverlapKatz similarityAll baselines (a combination of the first 5 heuristics in a 5-dim embedding)For practitionersWhen used as an API, EvalNE provides functions to:Load and preprocess graphsObtain general graph statisticsConveniently read node/edge embeddings from filesSample nodes/edges to form train/test/validation setsDifferent approaches for edge sampling:Timestamp based sampling: latest nodes are used for testingRandom sampling: random split of edges in train and test setsSpanning tree sampling: train set will contain a spanning tree of the graphFast depth first search sampling: similar to spanning tree but based of DFSNegative sampling or generation of non-edge pairs using:Open world assumption: train non-edges do not overlap with train edgesClosed world assumption: train non-edges do not overlap with either train nor test edgesEvaluate LP, SP and NR for methods that output:Node EmbeddingsNode-pair EmbeddingsSimilarity scores (e.g. the ones given by LP heuristics)Implements simple visualization routines for embeddings and graphsIncludes NC evaluation for node embedding methodsProvides binary operators to compute edge embeddings from node feature vectors:AverageHadamardWeighted L1Weighted L2Can use any scikit-learn classifier for LP/SP/NR/NC tasksProvides routines to run command line commands or functions with a given timeoutIncludes hyperparameter tuning based on grid searchImplements over 10 different evaluation metrics such as AUC, F-score, etc.AUC and PR curves can be provided as outputIncludes routines to generate tabular outputs and directly parse them to Latex tables(back to top)InstalationThe latest version of the library (v0.4.0) has been tested onPython 3.8.EvalNE depends on the following packages:NumpyScipyScikit-learnMatplotlibNetworkXPandastqdmkiwisolverBefore installing EvalNE make sure thatpipandpython-tkpackages are installed
on your system, this can be done by running:sudoapt-getinstallpython3-pip
sudoapt-getinstallpython3-tkOption 1:Install the library using pip:pipinstallevalneOption 2:Cloning the code and installing:Clone the EvalNE repository:gitclonehttps://github.com/Dru-Mara/EvalNE.gitcdEvalNEDownload dependencies and install the library:# System-wide installsudopythonsetup.pyinstall# Alternative install for a single userpythonsetup.pyinstall--userCheck the installation by runningsimple_example.pyorfunctions_example.pyas shown below.
If you have installed the package using pip, you will need to download the examples folder from
the github repository first.cdexamples/
pythonsimple_example.pyNOTE:In order to run theevaluator_example.pyscript, the
OpenNE library, PRUNE and Metapath2Vec are required. The instructions for installing
them are availablehere,here,
andhere,
respectively. The instructions on how to run evaluations using.inifiles are
provided in the next section.(back to top)UsageAs a command line toolThe library takes as input an.iniconfiguration file. This file allows the user
to specify the evaluation settings, from the task to perform to the networks to use,
data preprocessing, methods and baselines to evaluate, and types of output to provide.An exampleconf.inifile is provided describing the available options
for each parameter. This file can be either modified to simulate different
evaluation settings or used as a template to generate other.inifiles.Additional configuration (.ini) files are provided replicating the experimental
sections of different papers in the NE literature. These can be found in different
folders underexamples/replicated_setups. One such configuration file isexamples/replicated_setups/node2vec/conf_node2vec.ini. This file simulates the link
prediction experiments of the paper "Scalable Feature Learning for Networks" by A. Grover
and J. Leskovec.Once the configuration is set, the evaluation can be run as indicated in the next
subsection.Running the conf examplesIn order to run the evaluations using the providedconf.inior any other.inifile, the following steps are necessary:Download/Install the methods you want to test:Forconf.ini:InstallOpenNEInstallPRUNEFor other.inifiles you may need:Deepwalk,Node2vec,LINE,Metapath2vec, and/orCNE.Download the datasets used in the examples:Forconf.ini:StudentDBFacebook(combined network)ArXiv GR-QCFor other.inifiles you may need:Facebook-wallpostArXiv Astro-PhArXiv Hep-PhBlogCatalogWikipediaPPISet the correct dataset paths in the INPATHS option of the corresponding.inifile. And the correct method paths under METHODS_OPNE and/or METHODS_OTHER options.Run the evaluation:# For conf.ini run:python-mevalne./examples/conf.ini# For conf_node2vec.ini run:python-mevalne./examples/node2vec/conf_node2vec.iniNote: The input networks for EvalNE are required to be in edgelist format.As an APIThe library can be imported and used like any other Python module. Next, we
present a very basic LP example, for more complete ones we refer the user to theexamplesfolder and the docstring documentation of the evaluator and the split submodules.fromevalne.evaluation.evaluatorimportLPEvaluatorfromevalne.evaluation.splitimportLPEvalSplitfromevalne.evaluation.scoreimportScoresheetfromevalne.utilsimportpreprocessaspp# Load and preprocess the networkG=pp.load_graph('../evalne/tests/data/network.edgelist')G,_=pp.prep_graph(G)# Create an evaluator and generate train/test edge splittraintest_split=LPEvalSplit()traintest_split.compute_splits(G)nee=LPEvaluator(traintest_split)# Create a Scoresheet to store the resultsscoresheet=Scoresheet()# Set the baselinesmethods=['random_prediction','common_neighbours','jaccard_coefficient']# Evaluate baselinesformethodinmethods:result=nee.evaluate_baseline(method=method)scoresheet.log_results(result)try:# Check if OpenNE is installedimportopenne# Set embedding methods from OpenNEmethods=['node2vec','deepwalk','GraRep']commands=['python -m openne --method node2vec --graph-format edgelist --p 1 --q 1','python -m openne --method deepWalk --graph-format edgelist --number-walks 40','python -m openne --method grarep --graph-format edgelist --epochs 10']edge_emb=['average','hadamard']# Evaluate embedding methodsforiinrange(len(methods)):command=commands[i]+" --input{}--output{}--representation-size{}"results=nee.evaluate_cmd(method_name=methods[i],method_type='ne',command=command,edge_embedding_methods=edge_emb,input_delim=' ',output_delim=' ')scoresheet.log_results(results)exceptImportError:print("The OpenNE library is not installed. Reporting results only for the baselines...")pass# Get outputscoresheet.print_tabular()OutputThe library stores all the output generated in a single folder per execution. The name
of this folder is:{task}_eval_{month}{day}_{hour}{min}. Where{task}is one of:
lp, sp, nr or nc.The library can provide two types of outputs, depending on the value of the SCORES option
of the configuration file. If the keywordallis specified, the library will generate a
file namedeval_output.txtcontaining for each method and network analysed all the
metrics available (auroc, precision, f-score, etc.). If more than one experiment repeat
is requested the values reported will be the average over all the repeats.Setting the SCORES option to%(maximize)will generate a similar output file as before.
The content of this file, however, will be a table (Alg. x Networks) containing exclusively
the score specified in the MAXIMIZE option for each combination of method and network
averaged over all experiment repeats. In addition a second table indicating the average
execution time per method and dataset will be generated.If the option CURVES is set to a valid option then for each method dataset and experiment
repeat a PR or ROC curve will be generated. If the option SAVE_PREP_NW is set to True, each
evaluated network will be stored, in edgelist format, in a folder with the same name as the
network.Finally, the library also generates aneval.logfile and aeval.pkl. The first file
contains important information regarding the evaluation process such as methods whose
execution has failed, or validation scores. The second one encapsulates all the evaluation
results as a pickle file. This file can be conveniently loaded and the results can be
transformed into e.g. pandas dataframes or latex tables.ParallelizationEvalNE makes extensive use of numpy for most operations. Numpy, in turn,
uses other libraries such as OpenMP, MKL, etc., to provide parallelization. In order to allow for
certain control on the maximum number of threads used during execution, we include a simple bash
script (set_numpy_threads.sh). The script located inside thescriptsfolder can be given execution
permissions and run as follows:# Give execution permissions:chmod+xset_numpy_threads.sh# Run the script:sourceset_numpy_threads.sh# The script will then ask for the maximum number of threads to use.(back to top)ContributingContributions aregreatly appreciated. If you want to help us improve EvalNE,
please fork the repo and create a new pull request. Don't forget to give the project a star! Thanks!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestAlternatively, you can make suggestions or report bugs by opening a new issue with the appropriate tag
("feature" or "bug") and following our Contributing template.(back to top)LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)CitationIf you have found EvaNE useful in your research, please consider giving the repo a star
and citing ourpaper:@article{MARA2022evalne,title={EvalNE: A Framework for Network Embedding Evaluation},author={Alexandru Mara and Jefrey Lijffijt and Tijl {De Bie}},journal={SoftwareX},volume={17},pages={},year={2022},issn={100997},doi={10.1016/j.softx.2022.100997},url={https://www.sciencedirect.com/science/article/pii/S2352711022000139}}(back to top)