Package: assistant-improve-toolkit
Watson Assistant Improve NotebooksThis repository houses Watson Assistant Improve notebooks and the underlying assistant improve toolkit library.IntroductionTo help improving your Watson Assistant after you have deployed it to production, we prepared the following Jupyter notebooks. These notebooks include practical steps for measuring, analyzing, and actively improving your assistant in a continuous manner. Check outIBM Watson Assistant Continuous Improvement Best Practicesfor more details.Measure notebookcontains a set of automated metrics that help you monitor and understand the behavior of your system. The goal is to understand where your assistant is doing well vs where it isn’t, and to focus your improvement effort to one of the problem areas identified. This notebook generates an assessment spreadsheet for you to use to label problematic conversations, and then feed to the Effectiveness notebook.Effectiveness notebookhelps you understand the relative performance of each intent and entity as well as the confusion between your intents. This information helps you prioritize your improvement effort. The input to this notebook is an assessment spreadsheet generated from the Measure notebook. Update the marked columns in the spreadsheet with your labels and load it into the Effectiveness notebook for analysis.Logs notebookhelps you fetch logs using Watson Assistant API.  You can fetch logs with various filters, and save them as a JSON file, or export the utterances in the logs into a CSV file.  The JSON file can be loaded into the Measure notebook.  The CSV file can be used forintent recommendation service. Alternatively, you can run python scriptsfetch_logsandexport_csv_for_intent_recommendationto fetch logs and export them tointent recommendation CSV, respectively.  Runpython get_logs -handpython export_csv_for_intent_recommendation.py -hfor usage.Dialog Flow Analysis notebookhelp you assess and analyze user journeys and issues related to the dialog flow of ineffective (low quality) conversations based on production logs. Check outDialog Flow Analysisfor more details.Dialog Skill Analysis notebookhelp you analyze characteristics of your data such as the number of training examples for each intent or the terms which seem to be characteristic of a specific intent. Check outDialog Skill Analysisfor more details.Getting StartedYou can either run the notebooks locally or inIBM Watson Studio.Run locallyInstall Jupyter Notebook, seeJupyter/IPython Notebook Quick Start Guidefor more details.Download the Jupyter notebooks available in this repository'snotebookdirectory.Note: These notebook files are not designed for Watson Studio environmentStart jupyter serverjupyter notebookFollow the instructions in each of the notebooks. Be sure to add your Watson Assistant credentials if necessary.Run in Watson StudioCreate a Watson Studio account.Sign up inWatson Studio, or use an existing account. Lite plan is free to use.Create a new project and add a Cloud Object Storage (COS) account.For more information regarding COS plans, seePricing.CopyMeasureorEffectivenessnotebook from Watson Studio community into your project.Follow the instructions in each notebook to add project tokens and Watson Assistant credentials if necessary.GuidesLearn more about our measure and effectiveness notebook on Medium:Continuously Improve Your Watson Assistant with Jupyter NotebooksContributingSeeCONTRIBUTING.mdfor more details on how to contributeLicenseThis library is licensed under theApache 2.0 license.