Position: Data Engineer
Company: CHI Software
Experience required: 3y
Category: Data Science
English level: intermediate
Published: 2021-03-01T00:00:00+02:00

**Big Data**

• Experience managing big data.
• Be comfortable:
◦ Handling hundreds of TeraBytes of data in different databases across multiple services.
◦ Processing massive amounts of data in short bursts (~100 Billion items in span of ~2 days).
◦ Finding solutions to time-sensitive problems that will inevitably occur.

**Comprehensive AWS experience:**

• DynamoDB
◦ Experience handling DynamoDB when pushed well outside of the usual use-case/throughputs. Stackoverflow is of little use at this point, hands-on experience is needed.
◦ Understand how to take full advantage of partition/sort keys and secondary indexes (Global vs Local).
◦ Understand inner workings of the service and common pitfalls (hot-partitions, collection limits, etc)
◦ Understand throughput provisioning and its quirks (scaling limits per day, variable scale transition speeds, etc).
◦ Fully understand Dynamo pricing (there are some aspects that are not fully documented)
◦ Understand how table latencies vary depending on the activity and size of table.
• ElasticSearch
◦ Experience performing queries. Both for maintaining the current code and to be prepared for new features.
◦ Infrastructure knowledge: indexes, index aliases, shards, nodes and metric interpretation (indexing rate, search rate, search latency, etc.).
◦ Time-based indexes: SERPs cluster is adopting this paradigm. Doing queries, handling index rollovers (and potentially automating them) is important.
• S3
◦ Understand throughput limits and how to take full advantage.
◦ S3 as a data lake and how it interfaces with pipeline.
◦ Storage tiers and their pros and cons
• SQS
• EC2
• ECS
• Batch
◦ Be able to run thousands of parallel batch jobs.
• CloudWatch

**Architecture Design**

• Ability to create infrastructures that must:
◦ Handle big-data throughputs
◦ Scale
◦ Perform their tasks in reasonable amount of time
◦ Perform their task as cheaply as possible (it's easy to end up with five-digit bills if not careful)
• Understanding strengths and weaknesses of each service and know when to use which.
• Understand AWS services costs across the board.

**Nice to Have**

Would be good to have these complimentary skill set, we’ll need to figure out long term how to fulfill these areas even though ML / DS is not yet a full time roll.

**Machine Learning**

• Experience with optimization problems with complex requirements / customized training environments.
• ML on tabular data (NLP will most probably be neeeded at some point).
• Experience with Random Forest, GradientBoosting and NN classifiers/regressors.
• Experience with scikit-learn. PyTorch also appreciated.
• Understand performance metrics (Precision, Recall, F1-Score, MAPE, MSE, RMSE) and when to use them.

**Data Science**

• Needed for creating clean datasets for the ML models.
• Experience with Pandas or equivalent framework.
• Be able to manage noisy data and turn it into usable datasets.
• Google Search Console and Google Analytics.

**Industry experience / knowledge**

• Understand the SEO world
• Deep understanding of the domain in question (SEO metrics) is needed to be able to apply ML successfully.