Package: askbob
AskBobAskBobis a customisable framework for developing federated, privacy-safe voice assistants designed to be operated both: within IBM’s FISE ecosystem, aiming to combat social isolation, as a server integrated into the projects of teams 25 (concierge) and 38 (video conferencing); and interactively standalone on compatible low-power Windows and Linux desktop devices on which speech and data processing is performed locally to help safeguard users’ privacy.Its modular plugin architecture allows for voice assistant capabilities to be extended via additional third-partyskills pluginsinstallable at build-time, e.g. by interfacing with external services. Ask Bob is also accompanied with a Reactconfiguration generator web appto aid non-experts in designing new plugins and askills viewerweb app to inspect plugins installed on Ask Bob servers.The code in this repository was written byJeremy Lo Ying Ping.AskBobhas two primary modes of use:interactive mode (where users can interact directly with the voice assistant and hear audible responses)server mode (whereAskBobacts as a server responding to API calls)There is also an additional "voice-enabled" server option, which allows users to upload WAV files forAskBobto transcribe and interpret.WhenAskBobis installed locally on your system, both modes are available with proper configuration; however, only server mode (whether voiceless or voice-enabled) is available through Docker.Local use (interactive and server modes)InstallationFirst, ensure you have Python 3.7 andpipinstalled on your system.On Ubuntu,pipmay be installed using the following command:$curlhttps://bootstrap.pypa.io/get-pip.py|sudopythonOn Linux, you must ensure that you have the Python 3.7 dev package installed, e.g.python3.7-devon Ubuntu.Next, ensure that your versions ofpip,setuptoolsandwheelare up to date.$python-mpipinstall-UpipsetuptoolswheelDependencies shared across allAskBobutilisation modes (interactive mode, voiceless RESTful web API server or voice-enabled RESTful web API server) are installed by the defaultAskBobpackage wit no extras, as are the components required to runAskBobas a voiceless (i.e. with no speech-to-text capabilities) RESTful web API server.The latest release ofAskBobmay be installed from thePython Package Index(PyPI) using the following command:$python-mpipinstallaskbobAlternatively, although it is recommended thatAskBobbe installed fromPyPI, the very latest version ofAskBobcan also be installed from a cloned copy of this GitHub repository using the following commands (assuminggitis installed):$gitclonehttps://github.com/UCL-COMP0016-2020-Team-39/AskBob
$cdAskBob
$python-mpipinstall-e.TheAskBobPython package may be installed with multiple 'extras' depending on the use case as will be demonstrated below:voiceinteractivedocstestA compatiblespaCymodel must then be installed (theen_core_web_mdmodel is recommended) with the following command:$python-mspacydownloaden_core_web_mdUbuntu helper shell scriptTo aid users in installation, provided inscripts/install_voiceless_server.shis a shell script for Ubuntu that does the following:installsbuild-essentialinstallsgitinstallsAskBob(just the base dependencies needed to run as a voiceless server)installs theen_core_web_mdSpaCymodeltrainsAskBobbased off the plugins within thepluginsfolder and theconfig.inifileThis script must be copied to the root of anAskBobproject folderand executed there withsudoprivileges, e.g.$sudo./install_voiceless_server.shNote: the script relies that Python 3.7 is accessible in the terminal using the command wordpython. On some systems, this is not the case and it is insteadpython3or potentially evenpython3.7, so you may have to runalias python=python3prior to running the script.Voice-enabled RESTful web API server modeTo run theAskBobserver in voice-enabled mode, where WAV files of speech can be uploaded to theAskBobserver on the/voicequeryendpoint from whichAskBobwill produce a JSON response, additional voice-related dependencies must be installed.When installingAskBobfrom PyPi, use the following command:$python-mpipinstallaskbob[voice]When installingAskBobfrom a clone of this GitHub repository, instead use the following command:$python-mpipinstall.[voice]Once the extra dependencies are installed, amozilla\DeepSpeech-compatible model and external scorer must be downloaded into thedatafolder of anAskBobproject folder(with the runtimeconfig.inifile updated to reflect the correct filenames) in order forAskBobto use those models for speech transcription.Ubuntu helper shell scriptTo aid users in installation, provided inscripts/install_voice_server.shis a shell script for Ubuntu that does the following:installsbuild-essentialinstallsgitinstallsAskBobwith thevoiceextrainstalls theen_core_web_mdSpaCymodeldownloads a pre-trained English DeepSpeech model and external scorertrainsAskBobbased off the plugins within thepluginsfolder and theconfig.inifileThis script must be copied to the root of anAskBobproject folderand executed there withsudoprivileges, e.g.$sudo./install_voice_server.shNote: the script relies that Python 3.7 is accessible in the terminal using the command wordpython. On some systems, this is not the case and it is insteadpython3or potentially evenpython3.7, so you may have to runalias python=python3prior to running the script.Interactive modeTo runAskBobas a voice assistant interactively (i.e. with speech transcription and synthesis enabled), additional interactive mode-related dependencies must be installed.Firstly, the cross-platform audio APIportaudiomust be installed on your machine.On Ubuntu, theportaudiobinary can be installed with the following command:$sudoaptinstallportaudio19-devpython3-pyaudioTo useAskBobin interactive mode on Windows -- as this requires additional compilation -- you must haveBuild tools for Visual Studio 2019installed on your system (or any otherpip-compatible build tool). A direct download link may be found here:https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools&rel=16. You may then have to compile theportaudiobinary used byAskBobfrom source.Note: Christoph Gohlke maintains unofficial Windows binaries for Python extension packages, including forPyAudio, which may be installed usingpip install INSERT_BINARY_LOCATIONafter installing the relevant build tools and downloading thepyaudiowheel for Python 3.7.IfAskBobis being installed on macOS and you get anImportErrorrelated to_portaudio, then you may have to additionally run the following commands (assuming you havegitinstalled):$gitclonehttps://people.csail.mit.edu/hubert/git/pyaudio.git
$cdpyaudio
$sudopythonsetup.pyinstallWithportaudioproperly installed, you will then need to find amozilla\DeepSpeech-compatible model and scorer to be used withAskBobif you have not done so already. Once downloaded, place the files in thedatafolder of anAskBobproject folderand update the runtime configuration file (config.ini) with the correct file paths to the model and scorer, respectively.You may also have to modify the configuration depending on the text-to-speech voices made available by your operating system topyttsx3. These voices are listed whenAskBobis started in interactive mode and may also be found using the following sequence of Python commands:>>>importpyttsx3>>>print(*[voice.idforvoiceinpyttsx3.init().getProperty('voices')])HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens\TTS_MS_EN-US_DAVID_11.0HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens\TTS_MS_EN-GB_HAZEL_11.0HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens\TTS_MS_EN-US_ZIRA_11.0A shell script containing these Python commands is found atscripts/get_voices.shand is executable from this directory using the following command:$./scripts/get_voices.shOnce this is done, the additionalAskBobinteractiveextra must be installed. When installingAskBobfromPyPI, use the following command:$python-mpipinstallaskbob[interactive]When installingAskBobfrom a clone of this GitHub repository, instead use the following command:$python-mpipinstall.[interactive]IfAskBobis being run in interactive mode on a Linux system and the text-to-speech output fails to work, you may be missing a few additional system packages. On Ubuntu, these may be installed using the following command:$sudoaptupdate&&sudoaptinstallespeakffmpeglibespeak1TrainingTheAskBobvoice assistant must be trained before use using the following command:$python-maskbob--setup[optionaladditionalconfigurationJSONfile]Running the above command with no additional configuration JSON file will buildAskBobbased off only the installed plugins placed in thepluginsfolder:$python-maskbob--setupThe following command is an example of howAskBobcould be trained with such an additional build-time configuration JSON file:$python-maskbob--setupdefault_config.jsonMultilingual supportAskBobmay be used with any language supported for whichDeepSpeech(for speech-to-text),spaCy(for natural language processing) andpyttsx3(for text-to-speech) models exist. To do this, the relevant models must be downloaded and installed, and then theAskBobconfig.inimust be changed to reflect the use of the new models.First, download newDeepSpeechmodels into thedatafolder and adjust themodelandscorerparameters to use these new models:[Listener]model=data/deepspeech-0.9.1-models.pbmmscorer=data/deepspeech-0.9.1-models.scorerThen, download a newspaCymodel for the language and update the language configuration (e.g. for Frenchfr):$python-mspacydownloadfr_core_news_md[Rasa]language=frspacy_model=fr_core_news_mdModify thevoice_idparameter in the text-to-speech settings to that of an installed TTS voice for that language. Supported voices are listed whenAskBobis run in interactive mode.[TTS]voice_id=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens\TTS_MS_EN-GB_HAZEL_11.0UsageOnce installed on your system,AskBobshould be run from anAskBobproject foldercontaining at a bare minimum:/plugins- a folder of installed pluginsconfig.ini- a runtime configuration file (as described below)An example of askeleton project foldermay be found at ouraskbob-plugin-skeletonrepository:https://github.com/UCL-COMP0016-2020-Team-39/askbob-plugin-skeletonThisskeleton project repositorycan serve as a base for third-party developers to work on their ownAskBobplugins.All of the following commands must be executed from the base of anAskBobproject folder.AskBobmay be run interactively with the following command:$python-maskbobYou can specify your own runtime config.ini file with the -c flag:$python-maskbob-cconfig.iniAskBobcan also be run as a standalone server with the following command:$python-maskbob-sThe/voicequeryendpoint where a single-channel (preferably 16kHz) WAV file (<10MiB in size) may be uploaded is enableable by startingAskBobwith the-vflag in addition to the-sflag, i.e.$python-maskbob-s-vFurther help is available using the--helpflag.$python-maskbob--help
usage:__main__.py[-h][-cCONFIG][-wSAVEPATH][-fFILE][-dDEVICE][-rRATE][-s][-v][--setup[SETUP]]**AskBob**:acustomisablevoiceassistant.

optionalarguments:-h,--helpshowthishelpmessageandexit-cCONFIG,--configCONFIGTheconfigurationfile.-wSAVEPATH,--savepathSAVEPATHSave.wavfilesofutterencestoagivendirectory.-fFILE,--fileFILEReadfroma.wavfileinsteadofthemicrophone.-dDEVICE,--deviceDEVICEThedeviceinputindex(int)asgivenbypyaudio.PyAudio.get_device_info_by_index().Default:pyaudio.PyAudio.get_default_device().-rRATE,--rateRATETheinputdevicesamplerate(yourdevicemightrequire44100Hz).Default:16000.-s,--serveRunAskBobasaserverinsteadofinteractively.-v,--voiceEnablespeechtranscriptioninservermode.--setup[SETUP]SetupAskBobfromtheconfigurationJSONfileprovided.Docker (for use as a server only)AskBobmay be built and run as a server headlessly (with support for voice queries either enabled or disabled) using  Docker.The Docker Compose configurations contained withinthis repositoryare for building, testing, developing and contributing to theAskBobcodebase contained within this repository -- not creating your ownAskBobbuilds.Theplugin skeleton repositorycontains an example Docker configuration that you can use to runAskBobusing Docker within anAskBobproject foldercontaining your own personal project.Note: it is highly recommended that for your own personal projects, you do installAskBoblocally or use the example skeleton project Docker setup within anAskBobproject folder(containing the plugins and configuration specific to your project), rather than attempting to clonethis repositoryand develop your plugins there (while that may be possible).Installation & trainingIf you decide to use theDockerfileexamples provided in this repository (or the skeleton project folder repository), note that theAskBobis both installed and trained when the containers are built in one step.AskBobcan be built without support for voice queries withdocker-composeusing the following command:$docker-composebuildvoicelessSimilarly,AskBobcan be built with such support using the following command:$docker-composebuildvoiceAn additional build-time configuration (JSON) used in training may be specified using a build argument as in the following command:$docker-composebuild--build-argASKBOB_SETUP_CONFIG=default_config.jsonvoiceUsageTheAskBobserver can then be launched using the following commands:voice mode$docker-composeupvoicevoiceless mode$docker-composeupvoicelessDemo AskBob project folderAnAskBobproject folder for demonstrative purposes is found underexamples/demo. OnceAskBobis installed locally on your machine, you can change into that directory and run theAskBobdemonstration, e.g. as a server withpython -m askbob -s. If you download DeepSpeech model/scorer pair into thedatafolder (examples/demo/data) and update theconfig.inifile if the model/scorer filenames require updating, you may also runpython -m askbob -s -v. If the relevant dependencies for interactive mode are installed, the interactive demo may also be run withpython -m askbobfrom that folder.A Docker Compose configuration is also provided as a part of the demo project folder to run theAskBobas a server.AskBobcan be built without support for voice queries withdocker-composeusing the following command:$docker-composebuildvoicelessSimilarly,AskBobcan be built with such support using the following command (provided you add DeepSpeech models into adatafolder inexamples/demo):$docker-composebuildvoiceDeveloping new AskBob pluginsAnAskBobplugin consists of a plugin folder containing at a very minimum aconfig.jsonfile containing all of the data needed to train a Rasa model.AskBobis accompanied with aconfiguration generator web app(GitHub), which simplifies the drafting of theseconfig.jsonfiles to aid non-experts in designing newAskBobplugins.A specification of all the supported JSON options forconfig.jsonfiles may be found in thespecification.jsonfile within this repository.Plugins may also contain Rasa custom action code, which allowsAskBobplugins to run arbitrary Python code when certain spoken intents are triggered by the user (in either interactive or server modes).Files containing such custom action code must be listed within a Python list inside the__init__.pyfile at the root of the plugin folder in the following way (e.g. for a file containing custom action code calledactions.py):__all__=["actions"]Withinactions.py, custom action code takes the following form:fromtypingimportAny,Text,Dict,Listfromrasa_sdkimportAction,Trackerfromrasa_sdk.executorimportCollectingDispatcherimportaskbob.plugin@askbob.plugin.action("plugin_name","action_name")classActionHelloWorld(Action):defrun(self,dispatcher:CollectingDispatcher,tracker:Tracker,domain:Dict[Text,Any])->List[Dict[Text,Any]]:fromdatetimeimportdatetimedispatcher.utter_message(text="Hello, world")return[]More detail on Rasa custom actions may be found at theRasa SDK documentation.Plugin exampleAn example of a plugin that does not contain any custom action code may be found underexamples/plugins/main. If you are developing your own custom plugin, you may want to includeexamples/plugins/mainwithinpluginsin yourAskBobproject folder. It provides simple responses to "hello" and "goodbye", as well as an example on how to provide a natural language understanding fallback response triggered whenAskBobcannot determine a user's intent.An example of a plugin that does use custom action code is thetimeplugin found atexamples/plugins/time. It has the followingconfig.jsonfile contents:{"plugin":"time","intents":[{"intent_id":"ask_time","examples":["What time is it?","What time is it right now?","What time is it now?","Tell me the time","Tell me the time right now","Tell me the time now"]}],"actions":["fetch_time"],"skills":[{"description":"give the system time","intent":"ask_time","actions":["fetch_time"]}]}The__init__.pyfile withinexamples/plugins/timecontains__all__ = ["actions"]andactions.pycontains the following Python code:fromtypingimportAny,Text,Dict,Listfromrasa_sdkimportAction,Trackerfromrasa_sdk.executorimportCollectingDispatcherimportaskbob.plugin@askbob.plugin.action("time","fetch_time")classActionFetchTime(Action):defrun(self,dispatcher:CollectingDispatcher,tracker:Tracker,domain:Dict[Text,Any])->List[Dict[Text,Any]]:fromdatetimeimportdatetimedispatcher.utter_message(text="The time is{0}.".format(datetime.now().strftime("%H:%M")))return[]Further examples ofAskBobplugins may be found under theexamples/pluginsfolder.Plugin installationPlugins are installed by copying the plugin folder into thepluginsfolder of anAskBobproject folder.AskBobmust be retrained when new plugins are installed (python -m askbob --setup).A more visual way to view a summary of all the skills installed within a particularAskBobproject folderthan theGET /skillsraw JSON endpoint is to use theskills viewer web app(GitHub). RunAskBobin server mode (python -m askbob -s) and provide a URL to the skills endpoint to the web app, e.g.http://localhost:8000/skills.Runtime Configuration OptionsListener[Listener]model=data/deepspeech-0.9.1-models.pbmmscorer=data/deepspeech-0.9.1-models.scoreraggressiveness=1modelis the location to the DeepSpeech modelscoreris the optional location to an external DeepSpeech scoreraggressivenessis an optional integer (1 if unspecified) between 0 and 3 (inclusive) determining how aggressive the voice activity detector is at filtering out non-speech (0 is the least aggressive, 3 is the most aggressive).TTS[TTS]voice_id=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens\TTS_MS_EN-GB_HAZEL_11.0voice_idis the ID of the voice to be used by the pyttsx3 text-to-speech library. If this voice cannot be found, the default voice is used. You may need to install voices in a manner consistent with your operating system before this may work.Rasa[Rasa]config=data/rasa/configmodel=data/rasa/modelslanguage=enspacy_model=en_core_web_mdconfigis the location whereAskBobwill generate a set of Rasa YAML config filesmodelis the location whereAskBobwill place trained Rasa modelslanguageis a 2-letter language code representing the language to be used, e.g.enorfr.spacy_modelis the name of thespaCymodel to be used, e.g.en_core_web_mdIt is highly recommended that you do not change either of these values.Server[Server]host=0.0.0.0port=8000cors_origins=hostis the hostAskBobwill bind to when being run in server mode (0.0.0.0 if unspecified)postis the postAskBobwill bind to when being run in server mode (8000 if unspecified)cors_originsis the set of origins used for CORS (cross-origin resource sharing) - leave blank as above to disable CORS, otherwise something such ascors_origins = *would be typical for this configuration option.Plugins[Plugins]location=pluginsaction_server_port=5055summary=data/summary.jsonpluginsis the location of the plugins folder (this cannot contain any spaces and must be a valid python module name)action_server_portis the port used by the internal Rasa custom action server (5055 if unspecified)summaryis the location at which a summary of installed voice assistant skills is stored at build-timeIt is highly recommended that you do not change any of these values.Server endpointsThe following is a description of the endpoints available when runningAskBobin server mode.GET /This is just a landing page that returns the message below:"Hi, there! I think you might be in the wrong place... Bob."POST /queryRequest(encoded asx-www-form-urlencoded):sender- a unique identifier for the current usermessage- a query forAskBobto interpretResponse(JSON):{"query":"the original query","messages":[// Ask Bob messages]}For example, with thesenderas"askbob"and themessageas"tell me a joke",AskBobusing the examplepunsplugin could produce the following response:{"query":"tell me a joke","messages":[{"text":"One joke, coming right up!"},{"text":"Without geometry life is pointless."}]}In addition totext-type messages,AskBobplugins implementingRasacustom actions may also pass back arbitrary JSON ascustom-type messages, e.g.AskBobusing the FISE video conferencing integration plugin:{"query":"call John","messages":[{"text":"Calling John."},{"custom":{"type":"call_user","callee":"John"}}]}There are alsoimage-type messages available for third-party developers, which take the following form:{"image":"http://example.com/image_url.png"}GET /queryThis endpoint is identical toPOST /queryabove, other thansenderandmessagemay be passed as query string parameters, e.g.GET /query?sender=askbob&message=hello.GET /skillsThis endpoint produces a JSON index that describes of all the plugins installed on anAskBobserver. It returns data in the following format:{"plugins":[{"plugin":"miscellaneous","description":"","author":"","icon":""},{"plugin":"puns","description":"","author":"","icon":""}],"skills":[{"plugin":"miscellaneous","category":"miscellaneous","description":"Greet the user","examples":["Hello","Hi","Hey","Howdy","Howdy, partner"]},{"plugin":"miscellaneous","category":"miscellaneous","description":"Say goodbye to the user.","examples":["Goodbye","Good bye","Bye","Bye bye","See you","See you later","In a bit!","Catch you later!"]},{"plugin":"puns","category":"miscellaneous","description":"tell a joke","examples":["tell me a joke","tell me a joke please","tell me a dad joke","tell me a dad joke please","tell me a pun","tell me a pun please","give me a joke","give me a joke please","give me a dad joke","give me a dad joke please","give me a pun","give me a pun please","could you please tell me a joke?","could you tell me a joke?","I'd love to hear a joke","I'd love to hear a dad joke","I'd love to hear a pun","give me your cheesiest joke","give me a cheesy joke","tell me a cheesy joke","tell me something funny"]},{"plugin":"puns","category":"miscellaneous","description":"assure the user of the quality of dad jokes","examples":["Are you funny?","How funny are you?","How good are your puns?","How good are your dad jokes?","How cheesy are your jokes?"]}]}POST /voicequeryWhenAskBobis being run as a voice-enabled server, an additional/voicequeryendpoint becomes available.Request:sender- a unique identifier for the current user (encoded asx-www-form-urlencoded)speech- a 16000Hz single-channel WAV file under 10MiB containing a single speech utterance forAskBobto interpret (encoded as a form file upload)Response:as inPOST /queryandGET /queryDocumentationBefore attempting to generate the documentation, ensure that the documentation tools are installed.$python-mpipinstallaskbob[docs]The documentation can be generated using the following commands on Linux:$cddocs
$makehtmlOn Windows in a bash-style terminal, use themake.batfile provided in the following way:>cddocs
>./make.bathtmlThe generated documentation will be found atdocs/_build/html.In order to generate the full documentation, ensure all project dependencies are installed (all extras, i.e. voice, interactive, etc).TestsBefore running any tests, ensure thatAskBobwas installed with thetestextra to additionally include the testing suite.$python-mpipinstallaskbob[test]Tests (with code coverage report generation) may be run with the following command:$python-mpytesttests/The coverage report may be viewed afterwards by running the following command:$cdcov_test&&python-mhttp.server80And then navigating tohttp://localhost/in a browser.Note: the spaCyen_core_web_mdmodel must be installed to run the test suite, as well as all dependencies (including interactive mode dependencies).The DeepSpeech modeldata/deepspeech-0.9.1-models.pbmmand scorerdata/deepspeech-0.9.1-models.scorermust exist to run the test suite.