Position: Senior Data Engineer (Python) IRC106446
Company: GlobalLogic
Experience required: 3y
Category: Data Science
English level: intermediate
Published: 2020-12-01T00:00:00+02:00

Job Description
Must have:
Strong Python Knowledge
Strong knowledge of SQL, including Analytical Functions
Good understanding of differences between OLTP and OLAP workload
Good understanding of distributed data processing concepts
Experience with distributed data processing, e.g. Apache Spark, GCP DataFlow, etc.
Good knowledge of Cloud technologies for any Cloud provider
Some experience with ETL (or ELT)
At least a high-level understanding of Machine Learning methodologies, types and approaches

Nice to have:
Hands-on experience with AWS, namelly SQS\SNS, Redshift, EC2, S3
Knowledge of Apache Airflow
Experience with NoSQL distributed databases, especially with Cloud Warehouses (Amazon Reshift, Google BigQuery, Azure Synapse Analytics)

Job Responsibilities
Leading engineering group that responsible for development and deployment features to the platform that perfoms:

- getting, cleaning and save data from connected data feeds
- generate interim data aggregation flows that generates preliminary preparated data flows
- generate final data aggregates
- analytics generation ( reports, dashboard, etc. )
- МЛ моделей (которые разрабатываются другими) train ML models 
 improve data access interfaces and tools
-  managing data, i.e retention, backfilling, data managing, segmentation, performance tuning etc.

 
Department/Project Description
The project has the aim to develop AI Infrastructure (AII) is built atop Amazon Web Services resources with a particular focus on Redshift, SNS/SQS, CloudWatch, CloudFormation, Apache Airflow, as well as processing applications running on EC2 and other compute resources in the cloud. Software is developed in Python; the AWS Redshift database implements the Postgres API.

 

The first area will be developing, deploying, and designing monitoring for a large number of new data feeds that will be ingested into the Redshift database from internal and external data streams. The technology involved will be SNS/SQS, EC2 autoscaling cluster, AWS Redshift, CloudWatch. The code will be writing in Python, and CloudFormation will be used for deployment. Extensive unit and integrated testing, and continuous deployment will need to be improved or developed. The second area will focus on developing Airflow/Python script-driven SQL and Spark workflows that will utilize data stored in Redshift to generate appropriate downstream BI. The project will involve extensive SQL query optimizations, spinning up EMR clusters, exporting and importing data from Spark clustering, data cleaning, and validation. The final area will focus on improving continuous testing and deployment of our tools, and improving security in the AWS account