Position: Lead Data Engineer
Company: opinov8
Category: Data Engineer
Experience required: 5y
English level: upper
Published: 2023-03-01T00:00:00+02:00

The client owns an integrated data platform designed to help charterers, ship owners and operators make smarter decisions based on intelligent insights derived from reliable data. The platform brings together all owned and third-party data into a simple to use ecosystem powered by machine-learning that reduces fuel costs, improves vessel performance and helps cut and monitor greenhouse gas emissions.  
The team is committed to re-architecture toward cloud-based data processing solution into a generic data platform along with data pipelines and frontend optimization, which was focused on the determined business and architectural drivers, unification of cloud platform to Azure, and development of shared services. 

The team uses data engineering, data science, big data and full-stack engineering using technologies such as Python (Pandas/PySpark), Azure (mostly Cosmos DB, Airflow, Synapse, Data Factory), Databricks, Spark and Spark Streaming, public APIs, Docker, Terraform, C#/.NET, React.

As a Lead Data Engineer, you will be a key part of the team that owns our data platform and a key contributor to Opinov8 Data Community and Competence Center.
If you're excited by and have experience with ingesting, processing, storing, analyzing, and working in a Big Data environment, read on... 

**Responsibilities**
-Play a key role in architectural decisions as we build toward an efficient innovative analytical solutions using Big Data related technologies 
-Develop and launch new features to adapt to evolving business needs 
Build and modify Spark jobs (in Python & Pandas/PySpark) to perform various tasks 
-Deploy a range of cloud-based technologies for critical projects 
-Write, test, instrument, and deploy code to our Docker + Terraform environment 
-Help drive the innovation and evolution of the company products 
-Communicate with key technical stakeholders and third parties to understand requirements and help drive decisions, and gather requirements when underspecified 
-Build partnerships with Architects and key individuals within other functional groups 
-Actively participate in code review and test solutions to ensure it meets best practice specifications. Maintain high standards of code quality, and encourage the same by providing constructive code reviews to collaborators 
-Build a high-performance engineering culture, mentor team members of senior and middle+ level and provide them with the tools and motivation 
-Be an active and engaged owner of our data infrastructure 
-Be curious and seek to understand all aspects of our business 
-Troubleshoot and resolve issues, problems, and errors encountered across various systems 
-Collaborate with Data Science, Product, Research, and Engineering teams to iterate on the roadmap 
-Write technical documentation 
-Your participation and contributing to Opinov8 Data Community and Competence Center is a plus 
  
**Qualifications**
-7+ years of experience in software development, including at least 3 years of experience with writing in Python in containerized production environment 
-Ability to write high-performance, well-designed, well-crafted, readable, and tested code in a team environment 
-Good understanding of distributed computing principles, CI/CD principles and best practices 
-Solid experience with Spark & Spark Streaming (using Python), Databricks, HDFS and Azure (Cosmos DB), that includes streaming processing to joining and aggregating huge data sets, to integrating with third party data sources. Experience with debugging and optimizing Cloud-based data pipeline is must 
-Strong knowledge of SQL required 
-Solid understanding of both non-relational and relational database systems (Cosmos DB, DynamoDB, MongoDB, Cassandra) and familiarity with database fundamentals, such as ACID, database schema, normalized/denormalized data 
-Familiarity with the following Infrastructure technologies – Docker/Kubernetes/Yarn, Azure DevOps, Terraform 
-Computer Science degree or relevant experience 
-Clear and effective written and verbal communication skills 
-English: Upper-Intermediate (B2) and higher 
  
**Preferred Qualifications**
-Ability to build in multiple languages and the ability to pick up new languages quickly 
-Experience with maritime shipping business domain is a plus 
-Experience with real-time analytics (MS Power BI) is a plus 
-Compliance awareness, such as GDPR, SOX, HIPAA is a plus
-Relevant certifications as Security Specialist are considered an advantage
-Being motivated, persistent, independent, efficient and able to handle several projects; work under pressure with a solid sense for setting priorities is a plus 
-Ability to work in a fast-paced (startup like) Agile development environment is a plus