Position: Data Engineer (Spark OR Python OR R)
Company: Luxoft
Category: Data Science
Experience required: 3y
English level: upper
Published: 2020-12-01T00:00:00+02:00

**Project Description:**
The data42 Data pipeline Owner will be a core member of a data42 pipeline team supporting the program to provide the scientific community with quality, diverse, combined and secured clinical and research data that help to discover insights and contribute to the company mission. He manages expansion and optimization of our data and data pipeline architecture, as well as optimizes data flow and transformations.

Responsible for the implementation of data engineering and analytics to create new insights; implement analytical projects according to a new operating model which will be scaled up across the Company; highly transformational impact in terms of direct insights generated and new ways of working to create analytic results

Required technical and scientific skills:
- Computational/quantitative background; degree in a computational science (e.g. computer science, physics, engineering)
- Software engineering experience (versioning, Scrum, testing, collaboration good practices)
- Some experience with Python for pipelines, engineering practices (e.g. Python + Spark; Dask, Snakemake etc)
- Self-motivated, curious, proactive and accountable
- Strong independent problem-solving skills

Additional skills in two or more of the following areas are highly desirable:
- Strong experience with Python for pipelines, engineering practices (e.g. Python + Spark; Dask, Snakemake etc)
- Experience with R for data analysis (SparkR/sparklyr, tidyverse, mlr)
- Analysis of clinical trial datasets (SDTM/ADAM/...), incl. understanding of data formats, processes
- Analysis of Omics data (Genetic/RNA/other Omics) and bioinformatics databases and tools
- Experience building large scale software systems and implementation of creative solutions to difficult computational problems (with emphasis on performance and near real-time data analytics)
- Senior level data engineering (e.g. ETL) experience - Databricks or Apache Spark big data ecosystem experience - Proficient in SQL as well as Python, R or Scala
- Strong experience with AWS / Azure cloud technologies and stack

Other specific requirements could be:
- Experience in medical imaging solutions, medical image formats and metadata
- Understanding of statistical and machine learning, deep learning for structured and unstructured data
- Applied deep learning methods for image processing
- Familiarity with performance tuning skills (Spark, Data I/O); good knowledge about the internals of Spark and other components in Hadoop ecosystem (HDFS, Hive)" - Responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and transformations.
- Degree or higher degree in Life Sciences, Pharmacy, or Medicine (BSc, BA, PharmD, PhD, MD) and/or Degree in Computer Science with a specialization in medical data analysis

**Responsibilities:**
• Work closely with data scientists to solve analytical challenges involving large datasets, machine learning, deep learning in the areas of clinical, imaging or genomic data analysis.
• Analyze analytical patterns and blueprints to identify technical gaps and build best practices
• Design, create, test, and maintain optimal data pipeline architecture to ensure that it supports the requirements of the stakeholders. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources.
• Delivery of clear, maintainable, and well-tested code in a timely manner. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Create data-linked tools/scripts for data curators and data scientists as needed
• Collaboration with scientists, data managers, and technology teams to document and leverage their understanding of the data - Actively participate in agile work practices

Major KPIs:

1. Delivery of clear, maintainable, and well-tested code in a timely manner
2. 100% of data sources are available in the correct format before being needed for the knowledge graph

**Mandatory Skills:**
- Computational/quantitative background
- Degree in a computational science (e.g. computer science, physics, engineering)
- Software engineering experience (versioning, Scrum, testing, collaboration good practices)
- Some experience with Python for pipelines, engineering practices (e.g. Python + Spark; Dask, Snakemake etc)
- R Programming Language OR Apache Spark OR Python
- REST API
- SQL
- Self-motivated, curious, proactive and accountable
- Strong independent problem-solving skills
- Analysis of clinical trial datasets (SDTM/ADAM/...), including understanding of data formats, processes

**Nice-to-Have Skills:**
Additional skills in two or more of the following areas are highly desirable:
- Strong experience with Python for pipelines, engineering practices (e.g. Python + Spark; Dask, Snakemake etc)
- Experience with R for data analysis (SparkR/sparklyr, tidyverse, mlr)
- Analysis of Omics data (Genetic/RNA/other Omics) and bioinformatics databases and tools
- Experience building large scale software systems and implementation of creative solutions to difficult computational problems (with emphasis on performance and near real-time data analytics)
- Senior level data engineering (e.g. ETL) experience - Databricks or Apache Spark big data ecosystem experience - Proficient in SQL as well as Python, R or Scala
- Strong experience with AWS / Azure cloud technologies and stack

Other specific requirements could be:
- Experience in medical imaging solutions, medical image formats and metadata
- Understanding of statistical and machine learning, deep learning for structured and unstructured data
- Applied deep learning methods for image processing
- Familiarity with performance tuning skills (Spark, Data I/O); good knowledge about the internals of Spark and other components in Hadoop ecosystem (HDFS, Hive)" - Responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and transformations.
- Degree or higher degree in Life Sciences, Pharmacy, or Medicine (BSc, BA, PharmD, PhD, MD) and/or Degree in Computer Science with a specialization in medical data analysis