Package: nlubridge
Telekom NLU BridgeDevelopment•Documentation•Support•Contribute•LicensingThe goal of this project is to provide a unified API to several popular intent recognition
applications.About this componentInstallationThe core package including NLUdataset and Baseline vendors can be installed for
Python>=3.8 using pippip install nlubridgeNote that some vendors come with restrictions regarding the Python version, e.g. Rasa3
requires Python<3.11.To include optional dependencies for the vendors, e.g. Watson Assistant, typepip install nlubridge[watson]Following install options are available:watsonfasttextluisrasa2rasa3spacyhuggingfaceDevelopment tools can be installed with optiondevelop.Some vendors require access credentials like API tokens, URLs etc. These can be passed
on construction of the objects. Alternatively, such arguments can be passed as
environment variables, where the vendor will look for variables named variable
VENDORNAME_PARAM_NAME.Some vendors require additional dependencies. E.g., Spacy requires a model that
can be downloaded (for the  model de_core_news_sm) withpython -m spacy download de_core_news_smMigration from v0With realease 1.0.0 we introduce a couple of changes to the names of files and vendor
classes(see alsohttps://github.com/telekom/nlu-bridge/issues/18).Most notably:datasets.NLUdataset -> nlu_dataset.NluDatasetvendors.vendors.Vendor -> - vendors.vendor.Vendornew supackagedataloadersthat holds all functions for loading data into an NluDatasetnew functionnlu_dataset.concatto concatenate NluDatasets passed in a listcan load dataloaders, NluDataset, Vendor, OUT_OF_SCOPE_TOKEN, EntityKeys, concat,
directly from nlubridge likefrom nlubridge import VendorLoad vendors likefrom nlubridge.vendors import Rasa3formerTelekomModelnow calledCharNgramIntentClassifierSome vendor names changed for clarity and consistency (see "List of supported vendors"
for the new names)UsageHere is an example for the TfidfIntentClassifier:importosimportpandasaspdfromnlubridge.vendorsimportTfidfIntentClassifierfromnlubridgeimportNluDatasetdataset=NluDataset(texts,intents)dataset=dataset.shuffle()classifier=TfidfIntentClassifier()train,test=dataset.train_test_split(test_size=0.25,random_state=0)classifier=classifier.train_intent(train)predicted=classifier.test_intent(test)res=pd.DataFrame(list(zip(test.intents,predicted)),columns=['true','predicted'])If you need to configurestratification, use thestratificationparameter (defaults to"intents"and uses the intents in the dataset as stratification basis; whateverelseyou pass along has to conform tosklearn.model_selection.train_test_split(stratify=):train,test=dataset.train_test_split(test_size=0.25,random_state=0,stratification=None)# deactivate stratification (sklearn default for train_test_split)To compare your own vendor or algorithm to existing vendors in this package, you can
write a Vendor Subclass for your vendor, and possibly a dataloader function. Feel free
to share your implementation using this repo. Similarly, fixes and extensions for the
existing vendors are always welcome.LoggingMost of the code uses python logging to report its progress. To get logs printed out
to console or Jupyter notebook, a logger needs to be configured, before the nlutests
code. Usually, log messages are on INFO level. This can be configured like this:importlogginglogger=logging.getLogger()logger.setLevel(logging.INFO)logger.addHandler(logging.StreamHandler())Concepts / ArchitectureVendorsThevendorssubpackage implements standardized interfaces to
the specific vendors. A specificVendorinstance is in charge of dealing with
converting the data to the required format, uploading data to the cloud if applicable,
training models and making predictions.DatasetsThenlu_datasetmodule provides a standard interface to
NLU data. Data stored in different vendor's custom format can be loaded as a dataset
and provided to any different vendor.Data LoadersThedataloaderssubpackage provides functions to load
data that are in a vendor-specific format as NluDataset.List of supported vendorsVendor ClassStatusIntentsEntitiesAlgorithmTfidfIntentClassifier✓✓✗TFIDF on words + SVMFastText✓✓✗fasttextSpacy✓✓✗BoW linear + CNNWatsonAssistant✓✓✗Propietary (probably LR)Luisneeds testing✓✗Propietary (probably LR)CharNgramIntentClassifier✓✓✗tf-idf on char n-grams + SGDRasa2✓✓✓configurableRasa3✓✓✓configurableFeaturesAbstract class for Vendors with convenience methods (ex: scoring and scikit-learn compatibility)Abstract class for datasets with convenience methods (ex: train_test_split, indexing, iteration)Rate limiting to comply with cloud providers requirementsDevelopmentTBDBuildTBDCode of ConductThis project has adopted theContributor Covenantin version 2.0 as our code of conduct. Please see the details in ourCODE_OF_CONDUCT.md. All contributors must abide by the code of conduct.Working LanguageWe decided to applyEnglishas the primary project language.Consequently, all content will be made available primarily in English. We also ask all interested people to use English as language to create issues, in their code (comments, documentation etc.) and when you send requests to us. The application itself and all end-user facing content will be made available in other languages as needed.DocumentationThe full documentation for the telekom nlu-bridge can be found inTBDSupport and FeedbackThe following channels are available for discussions, feedback, and support requests:TypeChannelIssuesOther RequestsHow to ContributeContribution and feedback is encouraged and always welcome. For more information about how to contribute, the project structure, as well as additional contribution information, see ourContribution Guidelines. By participating in this project, you agree to abide by itsCode of Conductat all times.LicensingCopyright (c) 2021 Deutsche Telekom AG.Licensed under theMIT License(the "License"); you may not use this file except in compliance with the License.You may obtain a copy of the License by reviewing the fileLICENSEin the repository.Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See theLICENSEfor the specific language governing permissions and limitations under the License.