Position: Big Data Engineer
Company: United Software
Category: Data Engineer
Experience required: 5y
English level: upper

**Position Overview**

We are looking for a skilled Data Engineer to join our cloud solution team. The ideal candidate has an eye for building and optimizing the data stream and batch data processing system and will work closely with our systems architects, distributed computing engineers and ETL developers to help aggregate the data stream, direct the flow of data within the pipeline and ensure consistency of data delivery and utilization of reporting and data analytic systems.

**Responsibilities**

- Work closely with system architects and distributed computing engineers and reporting/data warehouse developers to create and optimize the architecture of the streaming data pipeline
- Design and build the streaming data pipeline to transform, aggregate and store the data
- Always angle for greater efficiency and robustness of the pipeline to support big volume of data stream
- Maintain the Azure DevOps environment to support CI/CD pipelines on hadoop/big data application and HPC windows pack/C# parallel computing platform
- Maintain and support the HDP/HPC Azure cloud environment
- Support the Azure Replication and DR environment
- Some possible work occasionally with C# and SQL Server.

**Qualifications**

- Graduate degree in Computer Science, Information Systems or equivalent quantitative field and 5+ years of experience in a similar Data Engineer role.
- Demonstrated ability to build processes that support data stream consumption, data transformation, data structures, metadata, dependency and workload management
- Strong interpersonal skills and ability to work with cross-functional teams
- Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Basic knowledge in network configuration, firewall and security, etc.
- Basic Hadoop Administration skills like:
       -  Managing / Monitoring the Hadoop cluster connectivity and 
          performance.
       -  Taking care of day to day running of Hadoop clusters.
       -  Manage and review the Hadoop Logs / Backup and recovery tasks
       -  Troubleshooting issues related to Hadoop Cluster.
- Experience with the following tools and technologies:
       - Big data experience, but preferred hadoop ecosystem distributions such as HDP, Cloudera etc. Cloudera is preferred
       - Public cloud such as Azure, AWS etc. Azure is preferred
       - Apache Hive
       - Apache Kafka
       - Spark-Streaming
       - Object-oriented/object function scripting languages such as Python, 
          Scala, C#.net etc.
       - Azure DevOps and CI/CD Pipeline
       - HPC Windows Pack and auto-grow/shrink
       - SQL Server, TSQL, etc…
       - C#.net