Position: Middle Data Engineer
Company: DevKit
Category: Data Engineer
Experience required: 3y
English level: upper

Job Description:

Whether our clients seek to revolutionize their company, establish credibility with a diverse array of stakeholders, or catalyze a profound cultural transformation, the most exceptional results arise from embracing a diverse spectrum of perspectives.

We employ methodologies that harness the depth and intricacy of human engagement, such as co-creative design, advanced data analytics, and visionary strategic planning.

Your Role:

As a Data Engineer, you'll play a pivotal role in shaping an emerging data ecosystem capable of hosting, analyzing, and modeling a diverse range of behavioral data. You'll become a crucial member of a newly formed team dedicated to the ambitious objective of crafting intelligence that enhances our understanding of, and ability to facilitate, human collaboration.

The ideal candidate will possess the capability to establish a novel data engineering infrastructure tailored for cutting-edge behavioral research. You'll collaborate closely with our tech team to implement high-quality data pipelines, integral to the success of our digital product. This role is particularly well-suited for those with an entrepreneurial mindset, as it offers an exciting opportunity to design data structures that capture intricate human behaviors. A minimum of five years of experience is preferred, and proficiency in NLP data modeling is essential for this position.

Responsibilities:

▶. Design and Implement Quality Pipelines: You'll take the lead in designing and implementing robust data pipelines, specifically focused on storing and delivering behavioral data, including natural language data for our client-facing product. Your work here will be essential in ensuring the reliability and efficiency of our data flow.
▶. Collaboration with Data Scientists and Project Managers: You'll work closely with our data scientists and project managers, adapting backend systems and data requirements to accommodate new research and development needs. Your input will be instrumental in aligning technical aspects with project objectives.
▶. ETL Automation and NLP Algorithms: You'll develop processes to automate Extract, Transform, Load (ETL) requirements, and you'll be responsible for crafting algorithms specifically tailored for Natural Language Processing (NLP) modeling. This automation will increase efficiency and data accuracy.
▶. Data Governance Frameworks: You'll collaborate in co-creating data governance frameworks, ensuring that data practices align with best standards across existing and new projects. This ensures consistency, security, and compliance.
▶. Technology Evaluation: As part of the technical team, you'll be at the forefront of evaluating and recommending new technologies that will become integral components of our products. Your insights will contribute to keeping our technology stack cutting-edge.
▶. Data Engineering Best Practices: You'll identify, document, and promote data engineering best practices within the team. Sharing knowledge and enforcing standards will elevate the overall quality of our work.
▶. Agile Environment: You'll comfortably operate in an Agile development environment, adapting to changing requirements and contributing to the iterative development process.
▶. Database ERDs, Pipeline Schemas, and Manuals: You'll create clear and comprehensive documentation, including database Entity-Relationship Diagrams (ERDs), pipeline schemas, and user manuals, ensuring seamless interaction for the technology team.
▶. Jira Ticket Ownership and Management: You'll own and manage data engineering-related Jira tickets during the development cycle, ensuring efficient progress and delivery.

We’d love to meet with you, if you have:

▶. Over 3 years of experience in diverse aspects of data engineering, you bring a wealth of expertise to the table.
▶. You've mastered Continuous Integration/Continuous Deployment (CI/CD) pipelines, protocols, and approaches, which is essential as our team rapidly expands. You're adept at working with Object-Relational Mapping (ORM) tools like SQLAlchemy and database management tools such as Alembic. Your hands-on experience in Natural Language Processing (NLP) is a valuable asset.
▶. You're skilled in mining structured, semi-structured, and unstructured data, and you excel in ETL/ELT programming. Your expertise extends to building and managing containerized environments using Docker and Kubernetes. You have a track record of constructing data warehouses with relational databases like Postgres and MySQL.
▶. Your experience shines through in crafting intricate pipeline orchestrations using Directed Acyclic Graphs (DAGs), demonstrated with tools like Airflow or AWS Step Functions. You've even developed data enrichment tools, leveraging web scraping and third-party APIs.
▶. Your proficiency with a wide array of AWS services is impressive, including S3, Lambda, Step Functions, RDS, ECR, ECS, Batch, EC2, EMR, and Glue. Python is your second language, and you're well-versed in essential language libraries such as SpaCy, nltk, pandas, wordnet, and Gensim.
▶. You hold a Bachelor's degree in Computer Science, Engineering, or a related field, or have equivalent training and work experience that speaks to your commitment. You're adept at monitoring and optimizing AWS costs related to ETL, demonstrating your understanding of the holistic process.

Nice-To-Haves:

▶. While these are the core aspects, your additional expertise is noteworthy. You're familiar with Tableau, and you have a grasp of NoSQL and graph-based database methods. Experience with Spark or PySpark adds depth to your skillset.
▶. Moreover, your familiarity with other AWS services, such as Neptune, CloudFormation, SQS, EKS, Kinesis, and Fargate, is valuable. Your exposure to MLOps integrations, including Databricks, MLFlow, Kubernetes, Kubeflow, and Snowflake, further demonstrates your versatility.
▶. Even the intricacies of video ETL pipelines, from storage to signal processing, don't intimidate you. Your comprehensive background makes you a key contributor to our team, bringing a unique blend of skills and passion to the forefront.

Interviewing stages:
Tech Interview, Welcome :)