Package: aiocorenlp
aiocorenlpHigh-fidelityasynciocapable StanfordCoreNLPlibrary.Heavily based onnerandnltk.Rationale and differences fromnltkFor every tag operation (in other words, every call toStanfordTagger.tag*),nltkruns a Stanford JAR (stanford-ner.jar/stanford-postagger.jar) in a newly spawned Java subprocess.
In order to pass the input text to these JARs,nltkfirst writes it to atempfileand includes its path in the Java command line using the-textFileflag.This method works well in sequential applications, however once scaled up by concurrency and stress problems begin to arise:Python'stempfile.mkstempdoesn't work very well on Windows to begin with and starts to break down under stress.Calls totempfile.mkstempstart to fail which in turn results in Stanford code failing (no input file to read).Temporary files get leaked resulting in negative impact on disk usage.Repeated calls tosubprocessmean:Multiple Java processes run in parallel causing negative impact on CPU and memory usage.OS-level subprocess and Java startup code has to be run every time causing additional negative impact on CPU usage.All this causes unnecessary slowdown and bad reliability to user-written code.Patchingnltk's code to usetempfile.TemporaryDirectoryinstead oftempfile.mkstempseemed to resolve issue 1 but issue 2 would require more work.This library runs the Stanford code in a server mode and sends input text over TCP, meaning:Filesystem operations and temporary files/directories are avoided entirely.There's no need to run a Java subprocess more than once.The only synchronization bottleneck is offloaded to Java'sSocketServerclass which is used in the Stanford code.CPU, memory and disk usage is greatly reduced.Differences fromnerasynciosupport.Method name manglingis inexplicably enabled in thener.client.NERclass, making subclassing not practical.The ner library appears to be abandoned.Differences fromstanzaasynciosupport.Stanza aims to provide a wider range of uses.Basic Usage>>>fromaiocorenlpimportner_tag>>>awaitner_tag("I complained to Microsoft about Bill Gates.")[('O', 'I'), ('O', 'complained'), ('O', 'to'), ('ORGANIZATION', 'Microsoft'), ('O', 'about'), ('PERSON', 'Bill'), ('PERSON', 'Gates.')]This usage doesn't require interfacing with the server and socket directly and is suitable for low frequency/one-time tagging.Advanced UsageTo fully take advantage of this library's benefits theAsyncNerServerandAsyncPosServerclasses should be used:fromaiocorenlp.async_ner_serverimportAsyncNerServerfromaiocorenlp.async_corenlp_socketimportAsyncCorenlpSocketserver=AsyncNerServer()port=server.start()print(f"Server started on port{port}")socket:AsyncCorenlpSocket=server.get_socket()whileTrue:text=input("> ")iftext=="exit":breakprint(awaitsocket.tag(text))server.stop()Context manager is supported as well:fromaiocorenlp.async_ner_serverimportAsyncNerServerserver:AsyncNerServerasyncwithAsyncNerServer()asserver:socket=server.get_socket()whileTrue:text=input("> ")iftext=="exit":breakprint(awaitsocket.tag(text))ConfigurationAs seen above, all classes and functions this library exposes may be used without arguments (default values).Optionally, the following arguments may be passed toAsyncNerServer(and by extensionner_tag/pos_tag):port: Server bind port. LeaveNonefor random port.model_path: Path to language model. LeaveNoneto letnltkfind the model (supportsSTANFORD_MODELSenvironment variable).jar_path: Path tostanford-*.jar. LeaveNoneto letnltkfind the jar (supportsSTANFORD_POSTAGGERenvironment variable, for NER as well).output_format: Output format. SeeOutputFormatenum for values. Default isslashTags.encoding: Output encoding.java_options: Additional JVM options.It is not possible to configure the server bind interface. This is a limitation imposed by the Stanford code.