Position: Data Engineer
Company: LitsLink
Experience required: 2y
Category: Data Science
English level: intermediate
Published: 2020-04-01T00:00:00+03:00

Some of the technologies we are using and are looking for:
• Cloud: AWS, Azure or GCP
• Languages: Python, Java, .NET, C++, JS (nice to have)
• Big Data: Hadoop, Spark, Hive
• Relational Database: MySql, PostgreSQL, Amazon Redshift
• NoSql: Redis, Elastic Search, Mongo DB
• Solid understanding of analytics solutions such as data lakes and data warehouses
• Experience with ETL and data transformation techniques
• Experience writing and optimizing SQL queries
• Deep understanding of statistical and Machine Learning techniques
• Basic knowledge in data sciences and its best practices
• Experience with machine learning libraries such as scikit-learn, pandas, NumPy, etc.
• Working in an agile environment, performing code reviews, operating in CI/CD environment

You are a great fit if you have:
• 2+ years of experience as a Data Engineer
• 2+ years of experience with production Hadoop pipelines and tools
• Experience with Spark and/or GCP Data Flow
• Practical experience with SQL and or NoSQL (MySQL|MongoDB|SQL Server|MemSQL).
• Ability to multi-task, prioritize and estimate effort
• Experience establishing integration standards and related processes.
• Good documentation habits
• Experience with workflow management tools (Jira, GitHub Workflow)
Even better if you have
• Self-supporting (able to set up and maintain dev environment)
• Experience with GCP Data Pipelines, Kubeflow
• Conceptual tool experience (UML Diagramming, Flow)
• Experience with containers and container-based architectures.
• Experience with R, Python and SQL
• Experience with Machine Learning algorithms & Apache Spark Dataframes|Datasets
• Experience working in an Agile environment

Core duties are:
• Design, code, create tests and integrate new features and functionality
• Design, build and productize complex data pipelines
• Learn the different AI/data science components/models in order for the algorithm to be properly translated into production code
• Participate in scrum project meetings and update stories using project management tools
• Apply CI/CD practices to prevent integration problems as well as ensure that the code is releasable at any point in time
• Participate in the estimation of the Stories based on defined Acceptance Criteria and Definition of Done
• Evolve and Maintain current Data Warehouse
• Implement ETL process (i.e. with Apache Spark or other tools)
• Lead Implementation and Recommend Evolution of Current Data Pipelines and Architectures
• Move data around to and from AWS and or GCP environments
• Monitor performance and recommend/implement any necessary infrastructure changes
• Define data retention and access policies
• Automate data pipeline infrastructure
• Design systems that are cost-aware and which provide the best value
• Manage cluster and environment and related scripts/tools
• Work with developers and other infrastructure engineers to deliver complementary solutions
• Work with a skilled team of developers and architects within a group dynamic setting
• Supporting the Architect in analyzing and recommending the best-suited architecture for our data pipelines
• Maintaining existing and creating new pipelines for our workloads
• Responsible for managing and overseeing Data Lake
• Improving bottlenecks around data processes through automation
• Providing data feeds to various stakeholders

What We Offer:
•  Competitive salary and high closing bonuses on market;
•  Flat organizational structure leading to fast decisions;
•  Opportunities for professional development and personal growth in share-knowledge culture;
•  Opportunity to relocate to the USA;
•  Business trips to clients, roadshows and international conferences (USA, Europe);
•  Large marketing and SEO / content department;
•  Comfortable conditions of work: modern green office, free breakfast and lunch, fruits, PlayStation, and a sports corner;
•  Flexible working hours and paid -> vacation, Orthodox holidays (15+days/year) and sick leave day scheme according to the corporate policy;
•  Corporate events, holiday celebrations, team building activities.

About project:
In this role, you will work in a client project environment with Product, Architecture and Applied Science Teams to contribute to the translation of complex AI/data science algorithms into scalable software, respectively to the envisioned Architecture to develop direct Business Value for our Products.