Position: Lead Data Engineer
Company: opinov8
Category: Data Engineer
Experience required: 5y
English level: upper
Published: 2023-02-01T00:00:00+02:00

The client owns an integrated data platform designed to help charterers, ship owners and operators make smarter decisions based on intelligent insights derived from reliable data. The platform brings together all owned and third-party data into a simple to use ecosystem powered by machine-learning that reduces fuel costs, improves vessel performance and helps cut and monitor greenhouse gas emissions.  

The team is committed to re-architecture toward cloud-based data processing solution into a generic data platform along with data pipelines and frontend optimization, which was focused on the determined business and architectural drivers, unification of cloud platform to Azure, and development of shared services. 

Outcomes: reliable cloud-based platform in Azure and all clients’ data migration to process data about 50 000 vessels all over the globe supporting procession with data quality and AI/ML enrichment of 7 million sensor readings per day. Solution provides rich analytics data to make data-driven business decisions. Customer leadership has a combined 100+ years in the maritime industry, that helps in enabling customers to get: 
•	Data is aggregated and validated for accuracy 
•	Teams are notified in real-time to impact savings today 
•	New and existing software is integrated into one solution 
•	Informed decisions are made based on critical KPI’s that look at entire operations 
•	Unparalleled transparency, visibility and accountability throughout entire organization.

Our approach to research and analytics allows us to capture all the data, gather it, draw actionable insights and determine how to best implement. 
We work with the biggest worldwide maritime shipping companies. 
Our team has a unique opportunity to work on data platforms, services and applications at scale, serving millions of shipments and tens of thousands of end-customers all over the globe. We're working on building an efficient and robust data platform and need Lead Data Engineer to help these millions of shipments to get their destinations in an optimal and environmentally safe way.  

Our stack is complex and we're looking for engineers who know how to write evolvable, properly instrumented, and efficient code as part of a growing distributed system.  
The team uses data engineering, data science, big data and full-stack engineering using technologies such as Python, Azure (mostly Cosmos DB, Airflow, Synapse, Data Factory), Databricks, Spark and Spark Streaming, public APIs, Docker, Terraform, C#/.NET, React. There are dozens of TBs of data in our Delta Lake, that is rapidly growing. This is no small responsibility: As a Lead Data Engineer, you will be a key part of the team that owns our data platform and a key contributor to Opinov8 Data Community and Competence Center. Data is our lifeblood, and there's a lot of it. If you're excited by and have experience with ingesting, processing, storing, analyzing, and working in a Big Data environment, read on... 

Responsibilities 
•	Play a key role in architectural decisions as we build toward an efficient innovative analytical solutions using Big Data related technologies 
•	Develop and launch new features to adapt to evolving business needs 
•	Build and modify Spark jobs (in Python & PySpark) to perform various tasks 
•	Deploy a range of cloud-based technologies for critical projects 
•	Write, test, instrument, and deploy code to our Docker + Terraform environment 
•	Help drive the innovation and evolution of the company products 
•	Communicate with key technical stakeholders and third parties to understand requirements and help drive decisions, and gather requirements when underspecified 
•	Build partnerships with Architects and key individuals within other functional groups 
•	Actively participate in code review and test solutions to ensure it meets best practice specifications. Maintain high standards of code quality, and encourage the same by providing constructive code reviews to collaborators 
•	Build a high-performance engineering culture, mentor team members of senior and middle+ level and provide them with the tools and motivation 
•	Be an active and engaged owner of our data infrastructure 
•	Be curious and seek to understand all aspects of our business 
•	Troubleshoot and resolve issues, problems, and errors encountered across various systems 
•	Collaborate with Data Science, Product, Research, and Engineering teams to iterate on the roadmap 
•	Write technical documentation 
•	Your participation and contributing to Opinov8 Data Community and Competence Center is a plus 
  
Qualifications 
•	7+ years of experience in software development, including at least 3 years of experience with writing in Python in containerized production environment 
•	Ability to write high-performance, well-designed, well-crafted, readable, and tested code in a team environment 
•	Good understanding of distributed computing principles, CI/CD principles and best practices 
•	Solid experience with Spark & Spark Streaming (using Python), Databricks, HDFS and Azure (Cosmos DB), that includes streaming processing to joining and aggregating huge data sets, to integrating with third party data sources. Experience with debugging and optimizing Cloud-based data pipeline is must 
•	Strong knowledge of SQL required 
•	Solid understanding of both non-relational and relational database systems (Cosmos DB, DynamoDB, MongoDB, Cassandra) and familiarity with database fundamentals, such as ACID, database schema, normalized/denormalized data 
•	Familiarity with the following Infrastructure technologies – Docker/Kubernetes/Yarn, Azure DevOps, Terraform 
•	Computer Science degree or relevant experience 
•	Clear and effective written and verbal communication skills 
•	English: Upper-Intermediate (B2) and higher 
  
Preferred Qualifications 
•	Ability to build in multiple languages and the ability to pick up new languages quickly 
•	Experience with maritime shipping business domain is a plus 
•	Experience with real-time analytics (MS Power BI) is a plus 
•	Compliance awareness, such as GDPR, SOX, HIPAA is a plus
•	Relevant certifications as Security Specialist are considered an advantage
•	Being motivated, persistent, independent, efficient and able to handle several projects; work under pressure with a solid sense for setting priorities is a plus 
•	Ability to work in a fast-paced (startup like) Agile development environment is a plus 
  
Additional information 
•	You will work in a team with engineers mostly of Senior level 
•	Scrum of Scrums process followed 
•	The system is hosted/deployed on Azure 
•	Business trips are possible to Warsaw, Poland 
•	There is an ongoing project, it is on the active development stage, and we plan this to be long-term 
•	You will be provided with personal development program and support that will allow you to be valued for your strengths 
•	Wide range of professional trainings and workshops 
•	Partial coverage of costs for certification and IT conferences 
•	Free English classes with certified teachers 
•	Variety of projects and possibility to change the project over the time 
•	Collaborate in a multicultural environment and exchange best practices with colleagues around the World 
•	Varied social benefits 
•	Remote work 
•	Flexible work hours 
•	Friendly team and enjoyable working environment 
•	Competitive compensation, regular assessments, and salary reviews