Position: Senior Data Engineer
Company: AgileHire
Category: Data Engineer
Experience required: 3y
English level: intermediate
Published: 2022-01-01T00:00:00+02:00

Customer company is a subscription billing and revenue management platform powering some of the fastest-growing brands around the world today, including Calendly, Hopin, Pret-a-Manger, Freshworks, Okta, Study.com, and others. Thousands of SaaS and subscription-first businesses process over billions of dollars in revenue every year through the this platform.
Headquartered in San Francisco, CA, our 500+ team members work remotely throughout the world, including India, the Netherlands, Paris, Spain, Australia, and the US.  Platform has raised over $230 million in capital and is funded by Accel, Tiger Global, Insight Partners, Steadview Capital, and Sapphire Ventures. We’re on a mission to push the boundaries of subscription revenue operations - not just ours, but every customer and prospective business with a recurring revenue model.

We are looking for a savvy Data Engineer to join our team. You'ill be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. 
The ideal candidate is an experienced data pipeline builder and data wrangler whoenjoys optimizing data systems and building them from the ground up. The right candidate willbe excited by the prospect of optimizing or even re-designing our company’s data architecture to support our next generation of products and data initiatives.

Roles and Responsibilities
? Create and maintain optimal data pipeline architecture
? Create and maintain events/streaming-based architecture/design
? Identify, design, and implement internal process improvements: automating manual
processes, optimizing data delivery, re-designing infrastructure for greater scalability,
etc.
? Build the infrastructure required for optimal extraction, transformation, and loading of
data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
? Create data tools for analytics and data scientist team members that assist them in
building and optimizing our product into an innovative industry leader.
? Work with data scientists to strive for greater functionality in our data systems.

Skills and Experience
? Experience building and optimizing ‘big data’ data pipelines, architectures and data
sets.
? Strong analytic skills related to working with structured/unstructured datasets.
? Build processes supporting data transformation, data structures, dimensional
modelling, metadata, dependency, schema registration/evolution and workload
management.
? Working knowledge of message queuing, stream processing, and highly scalable ‘big
data’ data stores.
? Experience supporting and working with cross-functional teams in a dynamic
environment.
? We are looking for a candidate with 4+ years of experience in a Data Engineer role.

They should also have experience using the following software/tools:
o Experience with big data tools: HDFS/S3, Spark/Flink, Hive, Hbase,
Kafka/kinesis etc.
o Experience with relational SQL (MySQL/Postgres) and NoSQL databases
(Cassandra, MongoDB, DynamoDB, HBase)
o Experience with key/value stores like Redis and search engines like
Elasticsearch.
o Experience with data pipeline and workflow management tools like Airflow etc.
o Experience with any public cloud like AWS services
o Experience with stream-processing systems: Spark-Streaming/Flink etc.
o Experience with object-oriented/object function scripting languages: Java,
Scala, Pyspark etc.