Package: lingo-fit
aws-ml-inference-demowe aim at building a product that can truly teach languages to our learners.
To do this we need to provide a personalized experience that suits each learner's needs.
As part of this personalized learning, imagine that we want to
prompt learners (within the app or via notifications)
with a personalized message to encourage them to review the material they have learned recently.
The message is personalized in the sense that it
recommends one of the Listening, Speaking, Writing, or Flashcards exercise types
that will most likely be attempted by the learner.
For the sake of this exercise assume the content and every other aspect of the messages are fixed and static.API and Architecture Design:For our service, we need to expose two endpoints for training and inference.
The incoming request will accept a learner identifier UUID
and respond with one of four possible values: "listening", "speaking", "writing", or "flashcard".Scale of the Service:
There are around 1 million active daily users concentrated mainly in North American and Europe.
This number might vary since we have 100k new app downloads every day.Assumptions on the regions and user base distributionThere are 8 regions with 24 availability zones in Europe
and 7 regions in North America
We assume there is negligible number of learners in elsewhere.With daily 100k downloads,
the active daily users number can rise up to 15 millions in 3-4 months if they start learning after download.
So the maximum (unlikely to reach) is 1 million user in every AWS geographical region.
In order to keep network overhead low, we aim to deploy an AWS Sagemaker endpoint to each of 15 regions.Sise of user records tableWe want to keep network overhead costs low to increase payload of the messages in the future or their granularity.
We expect each user to have 2 lessons (one hour maximum duration) per day maximum with maximum 6 exercsies per each lesson.
So each user requires maximum 12 recommendations per day. That makes 12 million recommendation in any 2 hour window
during the day.endpointsWith these assumptions we deploy 15 AWS sagemaker lambda functions into each of the 15 global regions.
The traffic between zones is rerouted automatically to the step function.We will have two API endpoints:POST /inference: This endpoint will accept a learner UUID in the request body and respond with a recommendation for an exercise type based on the Thompson Sampling algorithm.

POST /training: This endpoint will accept a learner UUID and the learner's response in the request body. The Lambda function will use the response to update the underlying models in the Thompson Sampling algorithm.About trainingThe training in batches will result in the same sequence of prior param values as
event-based training on new user interactions with the app in the region
There are many ways to distribute training but the methodology is still in research phase [1].
There, we adopt on event training for the bandits which simply amounts to updating the priors for the best arm.
We do not use the updated priors in the production endpoint.API design: Syncing model updatesEvery day, we shall update the model with latest priors for the day
This is the structure of the bucket
the structure of the prior files
"s3://personalization-service/priors/region=us-east2/timestamp=2022-01-01 T08:04:15.1256/priors.json"
the structure of the user experience data used for updating the priors
"s3://personalization-service/experience/region=us-east2/timestamp=2022-01-01 T08:04:15.1256/data.parquet
each local AWS sagemaker instance shall select latest by time priors for its region and re-run cloud build job to
update the endpointstorage size per regionEach request response is few bytes, 10 bytes, so overall data exchange in network is 0.012 Gb
we have around 10 columns of user activity in app that we record, it makes 40 bytes per record
Each exercise completion generates a record in an sql table, we expect
40 bytes * 12 million recommendation request 0.48 Gb per day of new data.
We might store last 30 days of user interaction records, that will be 14 Gb per regioningesting user experience dataAs soon as the learner studied 1/4, 2/4, 3/4, 4/4 of the lesson, the AWS Lambda function
gets triggered and calls Kinesis Data Stream, a possible events.json is attachedKinesis Data stream calls the Firehose and store the file on S3 bucket.As soon as the file upload to the bucket, another Lambda gets trigger and call the Glue Job.As soon as the file upload to the bucket, we trigger "train" endpoint to produce new priorsAWS Glue job runs and convert the JSON file format to Parquet file formatGlue Job place this new file to the destination bucket.SnowPipe configure on the destination bucket and trigger once file uploads to Bucket.Data gets ingest to the Snowflake.Modelling Problem:We want to model probabilities that the suggested exercise type is useful for the user and he/she will start and finish the exercise.
This probability shall be represented by the Beta-Bernoulli distribution
We want to model the relevancy of each of the exercises as a multi-armed bandit with one arm corresponding to suggested exercise type.
The probability of the completion of any of the four exercise types is represented by the beta bernoulli model.
In other words, for each user uuid + possible attached features vector of features is classified by four labels: "listening", "speaking", "writing", or "flashcard",
where each label indicates that exercise of this type will be completed by the user
The reward is modelled by the beta-bernoulli model.BillingSnowflake instance12 million requests per day per region
0.2 per 1M requests
0.2 * 12 = 2.4 usd per regionSnowflake instance incurs costs for storage of up to 14Gb per region
and compute on demand (runing sql queries against the data). We do not plan to run SQL queries every day.Bibliographyhttps://arxiv.org/pdf/2105.10590.pdf