Package: babelnet
This package consists of a Python API to work with BabelNet, a very large multilingual semantic network. For more
information, please refer to the documentation below on how to use the software, and our
website (https://babelnet.org) for news, updates and papers.Version compatibilityBabelNet Python API can be used with BabeNet 4.0 and above.ConfigurationAfter the installation, the first step to take when you want to use BabelNet in another project (or in the REPL) is to
create a file calledbabelnet_conf.ymlin the current working directory.
Alternatively, the path of the configuration file can be specified using theBABELNET_CONFenvironment variable.The content of thebabelnet_conf.ymlshould vary according to the usage mode of choice:Online Mode: uses the online REST service to retrieve the data. To use this mode you need an internet connection
and avalid API key.RPC Mode: reads data directly from a local copy of the BabelNet indices, making it more suitable for heavy
workloads than the online mode since it is faster and doesn't have usage limits. To use this mode you need theBabelNet indicesandDocker installed in your system.
TheRPC server controller(see below) requires additional dependencies that can be installed with the following pip command:pipinstallbabelnet[rpc]Further details on how to use these modes are provided in the following sections.Online ModeThis is the simplest version to use, since it requires onlya valid API key.
However, the drawback is that the iterators are unavailable, i.e. theiterator,offset_iterator,lexicon_iteratorandwordnet_iteratormethods.Assuming you have received by e-mail the key3x54mp13-8au0-o97q-9vzz-3vakcpec8w4p, add the following line
tobabelnet_conf.yml:RESTFUL_KEY:'3x54mp13-8au0-o97q-9vzz-3vakcpec8w4p'This will automatically be used to authenticate you on the official BabelNet REST service.The supported REST endpoints are:https://babelnet.io/v9/servicefor BabelNet 5.3(default)https://babelnet.io/v8/servicefor BabelNet 5.2https://babelnet.io/v7/servicefor BabelNet 5.1https://babelnet.io/v6/servicefor BabelNet 5.0https://babelnet.io/v5/servicefor BabelNet 4.0If you want to use a different REST endpoint, add the following line tobabelnet_conf.yml:# BabelNet 5.3 REST endpointRESTFUL_URL:'https://babelnet.io/v9/service'RPC ModeTo use the RPC mode you need alocal copy of the BabelNet indices. To download them, follow the procedure
on theofficial website.
This can be considered afullmode, because it has no usage limit and faster responses.BabelNet Python API requires PyLucene, which has a dependency on Lucene itself. The installation process of Lucene can be tricky
since it has many dependencies that need compiling. Because of this, we moved this PyLucene build and install process to
a simple Docker image. In the RPC mode, the Remote Procedure Call paradigm is applied in calling this Docker container
as a remote service, effectively decoupling PyLucene and BabelNet.To configure the APIs in RPC mode, you just need to add one of these lines to yourbabelnet_conf.yml, depending on
which protocol you want to use.The default protocol used by the RPC server is TCP. You can specify the URL where the server is listening with the
following configuration line.# TCP URL exampleRPC_URL:"tcp://127.0.0.1:7790"If the RPC server has the optional IPC protocol enabled, you can use it with the following configuration line.# IPC URL exampleRPC_URL:"ipc:///home/user/your_ipc_dir/socket"Important:to use lambdas in RPC mode, the client code must be run using thesame Python versionof the
server, i.e. Python 3.8, and thesame (or older) version of cloudpickle, i.e. 2.1.0.To start the server, you can either use theRPC server controlleror manually start the Docker.
In any case you needDocker to be installed in your system.
The controller is described in the following section; for details on how to directly use the
Docker image, please follow the documentation on theDocker Hub page.Note:when you update the API to a newer version, you need to either restart the server using the controller or
pull the new docker from the hub and start a new server with the updated image.RPC server controllerTo simplify the management of the RPC server, you can use thebabelnet-rpccommand.The additional dependencies required by the controller can be installed with the command:pipinstallbabelnet[rpc]For Windows users:if you are working in an Anaconda environment, you need to installpywin32using anaconda
with the following command:condainstallpywin32=227DocumentationOnce the server is started, the documentation of the Python API will be available athttp://localhost:7780,
or alternatively to the port defined by the arguments of thestartcommand.Start the serverTo start the server, you can use the commandbabelnet-rpc start.
If no arguments are provided, it will start in interactive mode, in which you will be prompted to provide the required
values.$babelnet-rpcstartBabelNet indices path: /home/user/BabelNet-5.3Port for documentation ([7780], -1 to ignore): 8080RPC mode ([tcp]/ipc/all): allPort for TCP mode ([7790]):IPC directory: your_ipc_dirStarting server...Server startedBabelNet Python API documentation is available at http://localhost:8080To use BabelNet in RPC mode, add one of these lines in your babelnet_conf.yml fileRPC_URL: "tcp://127.0.0.1:7790"RPC_URL: "ipc:///home/user/your_ipc_dir/socket"Alternatively, the values can be passed as arguments. The available arguments are:--bn <path>required, the BabelNet indices path--doc <port>port for the BabelNet API documentation (default7780)--no-docdisable the documentation port-m,--modethe RPC mode enabled on the server (tcp,ipcorall, defaulttcp).On Windows the only
available mode istcp.--tcp <port>the port for TPC mode (default7790)--ipc <path>the IPC directory (required with modeipcorall)--printprint the command instead of executing itExamples of usageBasic usage$babelnet-rpcstart--bn/home/user/BabelNet-5.3Starting server...Server startedBabelNet Python API documentation will be available at http://localhost:7790To use BabelNet in RPC mode, add this line in your babelnet_conf.yml fileRPC_URL: "tcp://127.0.0.1:7790"IPC mode without documentation$babelnet-rpcstart--bn/home/user/BabelNet-5.3--no-doc-mipc--ipcyour_ipc_dirStarting server...Server startedTo use BabelNet in RPC mode, add this line in your babelnet_conf.yml fileRPC_URL: "ipc:///home/user/your_ipc_dir/socket"Custom TCP port, print docker command$babelnet-rpcstart--bn/home/user/BabelNet-5.3--print--tcp1234To start the RPC server, run the following command:docker run -d --name babelnet-rpc -p 7780:8000 -p 1234:1234 -v "/home/user/BabelNet-5.3:/root/babelnet" babelscape/babelnet-rpc:latestBabelNet Python API documentation will be available at http://localhost:7780To use BabelNet in RPC mode, add this line in your babelnet_conf.yml fileRPC_URL: "tcp://127.0.0.1:1234"Stop the serverTo stop a running RPC server, run the command:babelnet-rpcstopCodeAssuming the installation and configuration phases have been completed, you can start working with BabelNet.The entry point in the library is thebabelnetpackage. It contains a set of functions that query the available
content. You can import the package by calling:importbabelnetasbnThe two main classes of BabelNet are:BabelSynset(a concept or named entity identified by a set of multilingual lexicalizations, each being a
BabelSense)BabelSense(a lexicalization of a given concept, i.e. a BabelSynset)For more details, see the API documentation athttps://babelnet.org/pydoc/1.2/.BabelSynsetABabelSynsetis a set of multilingual lexicalizations that are synonyms expressing a given concept or named entity.
For instance, the synset for car in the motorcar sense
lookslike this. After importingbabelnetasbnwe can use its functions to retrieve one or manyBabelSynsetobjects. For instance, to retrieve all the
synsets containingcarwe can callget_synsets:frombabelnet.languageimportLanguage# Given a word in a certain language,# returns the concepts (BabelSynsets) denoted by the word.byl=bn.get_synsets('car',from_langs=[Language.EN])We can also specify which of the parts of speech we are interested in and obtain only synsets for the specified part of
speech. In the following example, we retrieve all the verbal synsets containing the English lexicalizationrun:frombabelnet.languageimportLanguagefrombabelnet.posimportPOS# Given a word in a certain language and pos (part of speech),# returns the concepts denoted by the word.byl=bn.get_synsets('run',from_langs=[Language.EN],poses=[POS.VERB])Due to thenature of BabelNet, aBabelSynsetmay contain lexicalizations from different
sources. You can restrict your search only to your sources of interest. For instance:frombabelnet.languageimportLanguagefrombabelnet.posimportPOSfrombabelnet.data.sourceimportBabelSenseSource# Given a word in a certain language, returns the concepts# for the word available in the given sense sources.byl=bn.get_synsets('run',from_langs=[Language.EN],poses=[POS.NOUN],sources=[BabelSenseSource.WIKI,BabelSenseSource.OMWIKI])EachBabelSynsethas an ID that univocally identifies the synset, and that can be obtained via theidattribute of BabelSynset instances. If we have an ID and want to retrieve the corresponding synset, we can
useget_synset. For instance:frombabelnet.resourcesimportBabelSynsetID# Gets a BabelSynset from a concept identifier (Babel synset ID).by=bn.get_synset(BabelSynsetID('bn:03083790n'))returns the BabelSynset corresponding to
IDbn:03083790n, that is, the synset about BabelNet.If we want to retrieve the BabelSynset corresponding to a given WordNet 3.0 ID, we can do the following:frombabelnet.resourcesimportWordNetSynsetID# Gets the BabelSynsets corresponding to an input WordNet offset.by=bn.get_synset(WordNetSynsetID('wn:06879521n'))If we want to retrieve the BabelSynset corresponding to a given Wikidata page ID, we can do the following:frombabelnet.resourcesimportWikidataID# Gets the BabelSynsets corresponding to an input Wikidata page ID.by=bn.get_synset(WikidataID('Q4837690'))If we want to retrieve the BabelSynsets containing a given Wikipedia page title, we can use the functionget_synsets:frombabelnet.languageimportLanguagefrombabelnet.posimportPOSfrombabelnet.resourcesimportWikipediaID# Given a Wikipedia title, returns the BabelSynsets which contain it.byl=bn.get_synsets(WikipediaID('Men in Black (film 1997)',Language.IT,POS.NOUN))BabelSenseABabelSenseis a term (either word or multi-word expression) in a given language occurring in a certainBabelSynset. Each occurrence of the same term (e.g., car) in different synsets is, therefore, a differentBabelSenseof that term.Now let's look at the functions to retrieve aBabelSenseusing thebnmodule we have imported earlier:frombabelnet.languageimportLanguagefrombabelnet.posimportPOSfrombabelnet.data.sourceimportBabelSenseSource#  Returns the senses for the word in a certain language.senses1=bn.get_senses('run',from_langs=[Language.EN])# Returns the senses for the word in a certain language and Part-Of-Speech.senses2=bn.get_senses('run',from_langs=[Language.EN],poses=[POS.VERB])# Returns the senses for the word with the given constraints.senses3=bn.get_senses('run',from_langs=[Language.EN],poses=[POS.VERB],sources=[BabelSenseSource.WIKI,BabelSenseSource.OMWIKI])Once we have aBabelSense, we can go back to the synset it belongs with thesynsetproperty:by=sense.synsetWe can view theBabelSynsetas a container ofBabelSenses, i.e., the lexicalizations in the various languages
contained in the synset that express its concept or named entity.Some attributes of BabelSynset and BabelSenseWe are now going into details about important attributes (methods, properties)
of theBabelSynsetandBabelSenseclasses.BabelSynsetBabelSynsetis composed of various elements, which we describe below. Furthermore, aBabelSynsetis connected to
otherBabelSynsetobjects. The main components of aBabelSynsetare objects of the following types:BabelSense(a lexicalization of the concept, see above)POS(the synset's part of speech)BabelGloss(a definition of the concept in a given language)BabelExample(an example sentence of the meaning expressed by the synset)BabelImage(an image depicting the concept)BabelSynsetRelation(an edge semantically connecting the synset to another synset)Let's take a look at the main methods and properties of aBabelSynsetobject which we callby.Note:to
obtainBabelSynsetobjects we can also use the above examples.# Get a BabelSynset from a concept identifier (Babel synset ID).by=bn.get_synset(BabelSynsetID('bn:03083790n'))# Most relevant BabelSense to this BabelSynset for a given language.bs=by.main_sense(Language.EN)# The part of speech of this BabelSynset.pos=by.pos# True if the BabelSynset is a key conceptis_key_concept=by.is_key_concept# Gets the senses contained in this BabelSynset.senses=by.senses()# Collects all BabelGlosses in the given source for this BabelSynset.glosses=by.glosses()# Collects all BabelExamples for this BabelSynset.examples=by.examples()# The images (BabelImages) of this BabelSynset.images=by.images# Collects all the edges incident on this BabelSynset.edges=by.outgoing_edges()# Gets the BabelCategory objects of this BabelSynset.cats=by.categories()BabelSenseWe now have a look at the BabelSense attributes. The main components of a BabelSense are:BabelSynset(the synset the sense belongs to)POS(its part-of-speech tag)the lemma string (the lexicalization of the sense)BabelSensePhonetics(the written and audio pronunciations of this sense)BabelSenseSource(the source of the sense, e.g.: Wikipedia, WordNet, etc.)Some code retrieving the above information follows:bs=by.main_sense(Language.EN)# The language of this BabelSenselang=bs.language# The part-of-speech tag of this BabelSensepos=bs.pos# True if the BabelSense is a key conceptis_key_concept=bs.is_key_sense# The lemma of this BabelSenselemma=bs.full_lemma# The normalized lemma of this sense (i.e., lowercase, without parentheses, etc.)normalized_lemma=bs.normalized_lemma# The pronunciations of this sensepronunciations=bs.pronunciations# The source of the sense; ex: Wikipedia, WordNet, etc.source=bs.sourceUsage examplesHere we show full examples that show how you can use the BabelNet API to accomplish several tasks.Retrieve all BabelSynset objects for a specific wordimportbabelnetasbnfrombabelnetimportLanguageforsynsetinbn.get_synsets('home',from_langs=[Language.EN]):print('Synset ID:',synset.id)For a specific word retrieves all BabelSynset objects in English, Italian and FrenchimportbabelnetasbnfrombabelnetimportLanguagesynsets=bn.get_synsets('home',from_langs=[Language.EN],to_langs=[Language.IT,Language.FR])forsynsetinsynsets:print('Synset ID:',synset.id)Retrieve all BabelSense objects for a specific BabelSynset objectimportbabelnetasbnfrombabelnetimportBabelSynsetIDsynset=bn.get_synset(BabelSynsetID('bn:00000356n'))# a synset is an iterator over its sensesforsenseinsynset:print('Sense: '+sense.full_lemma,'Language: '+str(sense.language),'Source: '+str(sense.source),sep='\t')phonetic=sense.pronunciationsforaudioinphonetic.audios:print('Audio URL',audio.validated_url)Retrieve all BabelSense objects for a specific Wikidata page idimportbabelnetasbnfrombabelnet.resourcesimportWikidataIDsynset=bn.get_synset(WikidataID('Q4837690'))# a synset is an iterator over its sensesforsenseinsynset:print('Sense: '+sense.full_lemma,'Language: '+str(sense.language),'Source: '+str(sense.source),sep='\t')phonetic=sense.pronunciationsforaudioinphonetic.audios:print('Audio URL',audio.validated_url)Retrieve Wikidata id for each BabelSense in a BabelSynsetimportbabelnetasbnfrombabelnetimportBabelSynsetID,BabelSenseSourceby=bn.get_synset(BabelSynsetID('bn:00000356n'))forsenseinby.senses(source=BabelSenseSource.WIKIDATA):sensekey=sense.sensekeyprint(sense.full_lemma,sense.language,sensekey,sep='\t')Retrieve neighbors of a BabelSynset objectimportbabelnetasbnfrombabelnetimportBabelSynsetID,Languagefrombabelnet.data.relationimportBabelPointerby=bn.get_synset(BabelSynsetID('bn:00015556n'))foredgeinby.outgoing_edges(BabelPointer.ANY_HYPERNYM):print(str(by.id)+'\t'+by.main_sense(Language.EN).full_lemma,edge.pointer,edge.id_target,sep=' - ')Retrieve the distribution of relationships (frequency of each BabelPointer type) for a specific wordfromitertoolsimportgroupbyimportbabelnetasbnfrombabelnetimportLanguagesynsets=bn.get_synsets('car',from_langs=[Language.EN])li=[edge.pointer.symbolforsynsetinsynsetsforedgeinsynset.outgoing_edges()]forp,lingroupby(sorted(li)):print(p,len(list(l)),sep='\t')MultithreadingInonline moderequests can come from different threads or processes and are elaborated concurrently.InRPC mode, using the API simultaneously from multiple threads is discouraged due to Python's threading
management and the limitations of the RPC library. Since sending concurrent requests to the server can lead to long
response times, to avoid timeouts it is recommended to use a limited pool like in the following example.importconcurrent.futuresfromdatetimeimportdatetimefromsysimportstdoutimportbabelnetasbnfrombabelnetimportLanguage# function called from the threadsdeffunc(name:str,word:str):stdout.write(datetime.now().strftime("%H:%M:%S.%f")+" - Start - "+name+"\n")synsets=bn.get_synsets(word,from_langs=[Language.EN])glosses=[]forsynsetinsynsets:gloss=synset.main_gloss(Language.EN)ifgloss:glosses.append(gloss.gloss)stdout.write(datetime.now().strftime("%H:%M:%S.%f")+" - End   - "+name+"\n")return{word:glosses}word_list=["vocabulary","article","time","bakery","phoenix","stunning","judge","clause","anaconda","patience","risk","scribble","writing","zebra","trade"]withconcurrent.futures.ThreadPoolExecutor(max_workers=8)asexecutor:future=[]fori,winenumerate(word_list):future.append(executor.submit(func,f'Thread{i}"{w}"',w))results={}forfinfuture:results.update(f.result())forw,gsinresults.items():forgings:print(w,g,sep='\t')AuthorsBabelscape
(info@babelscape.com)LicenseBabelNetand itsAPIare licensed under theBabelNet Non-Commercial License.