Package: otter-service
otter-nb serviceThis repo contains a tornado flask app that accepts .ipynb files and grades them in a dockerized environment. Assuming you are running a Jupyterhub, you can ask Jupyterhub to run this otter-service as a service; you also have the option to run it in a stand alone manner. Grades are saved to a gcloud Cloud Firestore.A separate Jupyterhub extension,otter_submit, presents a "Submit" button to the user in a notebook rendered in Jupyterhub. The button is configured to serialize and send the notebook to this otter-service as well as notify the the user of the successful submission.FireStore/Database setupAll grades are written to gcloud Cloud FireStore. During local testing with pytest, the tests remove the collection created after verifying the data was written. Local docker testing, does not delete the entries but the collection is called, otter-docker-local-test, and can be viewed and deleted by going to the gcloud console and navigating to Cloud Firebase or Cloud Firestore.ConfigurationNotebook(ipynb metadata)The ipynb notebooks need to include the metadata for which assignment they are. There are three pieces of information that are relevant: the course name, section and lab. They should be nested in an "otter_service" attribute as shown below. These values correspond to the course-config.json file described in the section below. These are set in the metadata section of every notebook:metadata:{
  "otter_service": {
    "course": "8x",
    "section": "1",
    "assignment": "lab01"
  }
  ...
}Course ConfigurationEach course needs to provide two pieces of information to whomever is handling otter-service(right now:sean.smorris@berkeley.edu) that is deployed in the secrets file of
this applicationA Github access key to the private materials repo.The name of repository where the autograder.zip files are kept. (e.g. github.com/data-8/materials-x22-private)Finally, the private repository with the autograder.zip files needs to contain a file named: course-config.json. The file is structured like this:(1) course name which matches the course name in every notebooks metadata(2) section which matches the section in every notebooks metadata - if only one section put '1'(3) Edx course id(4) subpath to each autograder.zip in private autograder.zip repo(5) each EDX assignment name and assignment Id -- the assignment name must match the notebook metadata{
  "8x" : { //(1)
    "1": { //(2)
      "course_id": "BerkeleyX+Data8.1x+3T2022", //(3)
      "subpath_to_zips": "materials/x22/lab/1", //(4)
      "assignments": {  //(5)
        "lab01": "6333e19d6b4d46f88df671ba50f616d8",
        "lab02": "fbf6740d45094b9b977111d218969273",
        "lab03": "0c8f28bdc48d4231843f62b512d73638",
        "lab04": "8db69daf14cf4751a088106be912c0cd",
        "lab05": "4cb9e3491c284cd5ae29bb48219ee15b"
      }
    },
    "2":{
      "course_id": "BerkeleyX+Data8.2x+3T2022",
      "subpath_to_zips": "materials/x22/lab/2",
      "assignments": {
        "lab01": "7abc0025c10f4b8ab123dbc88d34faaf",
        ...Here is an example of the folder structure in the materials-x22-private repo.Note how the folder structure is mirrored in the course-config.json.If you are not posting grades to an LTI server, than you do not need to worry about this.Test filesThis gets tricky. The notebooks and the corresponding test files used by this service are of course connected. The filesDockerfileandDockerfile-dev(used for local testing) download the current set of test files from the repository,materials-x22-private, for thematerials-x22notebooks. If you bring in different notebooks, you would need to change the two dockerfiles to bring in the corresponding tests.We assume a specific path to the test files. If you mirror the path found in thematerials-x22-privaterepository, all will work well. If you change the path, then you must change thesolutions-pathvariable ingrade_assignment.py.Docker ImageThis just FYI. The Dockerfile pulls an image :docker pull ucbdsinfra/otter-graderThis image is used by otter-grader to run the containerized grading.EdX/LTI integrationThe system posts the grade back to the EdX via LTI. You need to have theLTI_CONSUMER_KEYandLTI_CONSUMER_SECRETdefined and encoded viasopsfor this to work correctly. The secrets are inotter-service/secrets/gke_key.yamlExternal installation with a re-direct from JupyterhubThis is the current deployment configuration. We deploy the otter-service to gcloud and there is a re-direct from the Jupyterhubconfiguration filesin thedatahub repositorythat passes authentication information to otter-service.Once the GKE cluster is created in gcloud, executing thedeployment/cloud/deploy.shfile  deploys the service to the cloud.Deployment Details:Rollback:If we deploy and find problems the quickest way to rollback the deployment is to look at the revision history and undo the deployment by deploying to a previous revision number:kubectl rollout history deployment otter-pod -n grader-k8-namespacekubectl rollout history deployment otter-pod -n grader-k8-namespace --revision=# <-- to see details like the version of the image usedkubectl rollout undo deployment/otter-pod -n grader-k8-namespace --to-revision=#CI/CD:If you push a tag in the standard form of a version number(XX.XX.XX), GitHub action creates a release from this tag, pushes the release to pypi.org, builds the docker image, pushes it google's image repository and deploys the new image into the GKE cluster.pod size recommendationsThere is a vertical pod autoscaler deployed to recommend memory and cpu sizing to the otter-pod pods.
You can see recommendations via either of these commands:kubectl get vpa -n grader-k8-namespacekubectl get vpa -n grader-k8-namespace --output yamlIt is called an autoscaler but I configured the resource to just recommend and not actually autoscale vertically.pod horizontal scalingA horizontal autoscale is configured to spin up a new pod when 80% of CPU requested is utilized. There is maximum
of 10 pods allowed.You can see the status of the horizontal scaling via this command:kubectl get hpa -n grader-k8-namespacepytestRun ./deployment/local/pytest.sh -- this will start the Firestore emulator and run the tests.
If the emulator is already running it shuts it down. I shut down the emulator when the tests are done
as well but you could comment out this line to check out the data that was stored.Local installation for testing/developingInstall a FireStore Emulator so you test locally:Install FireStore CLI:https://firebase.google.com/docs/cli/#install-cli-mac-linuxfirebase loginfirebase projects:listfirebase setup:emulators:firestoremake java jdk installedfirebase emulators:start --only firestore --project data8x-scratchYou can see the UI here:http://localhost:4000/firestorepython3 -m pip install google-cloud-firestoreYou will notice the re-direct in firebase_local fixture used by test_write_grade in test_otter_nb.pyWith docker installed, you can use theDockerfile-devfile to deploy a local instance of otter-service. Thedeployment/local/build.shfile gives some guidance to building and installing local changes to otter-service for testing. The usual process is to make changes, executebuild.sh, which relies on adocker-compose.ymlfile. A sample is below but before we look, I would also study the filetests/integration.py. If you execute this file, you can test the service via a web connection.Sample docker-compose.yml:version: "3.9"
services:
  app:
    image: otter-srv
    build:
      context: .
      dockerfile: Dockerfile-dev
      args:
        OTTER_SERVICE_VERSION: whatever_version you specify in otter-service/__init__.py
    env_file:
      - ../.local-env
    ports:
      - 10101:10101
    volumes:
      - /tmp/otter:/mnt/data
    entrypoint: ''

networks:
  default:
    driver: bridgeNotes:.local-env These are environment variables that must be set. They mirror the variables indeployment/cloud/deployment-config-encrypted.yaml. You do not need to encrypt your local-env file with sops.Service installation in JupyterHubInstructions can be found here for running it as a service within yourjupyterhub