For the given job description <<Allergan Data Labs is on a mission to transform the Allergan Aesthetics beauty business at AbbVie, one of the largest pharmaceutical companies in the world. Our iconic brands include BOTOX Cosmetic, CoolSculpting, JUVDERM and more.

Allergan Data Labs is on a mission to transform the Allergan Aesthetics beauty business at Abbvie, one of the largest pharmaceutical companies in the world. Allergan Aesthetics brands include Botox, CoolSculpting, Juvderm and many more. The medical aesthetics business is ripe for disruption and we're building a high performing Data Science and Engineering team to do just that.

Our team has successfully launched a new and innovative technology platform, All, which serves millions of consumers, tens of thousands of aesthetics providers, and thousands of colleagues throughout the US. Since its launch in November 2020, All has delivered curated promotions, personalized experiences, and millions of consumers use it as part of their beauty journey.

We're looking for a Senior Data Engineer interested in working within a startup-oriented environment while having the backing of a very large company. If that's you, please read on.

You Will
Work with cross functional partners - Data Scientists, Engineers, Product Managers to understand and deliver data needs.Champion code quality, reusability, scalability, security and help make strategic architecture decisions with the lead engineerDesign, build and launch extremely efficient and reliable data pipelines to move data across a number of platforms including Data Warehouse, online caches and real-time systems.Build product-focused datasets and scalable, fault-tolerant pipelines.Build data quality checks, data anomaly detection, and optimize pipelines for ideal compute and storage.Develop APIs and Microservices to expose and integrate data products with software systems
Required Experience & Skills
5+ years of experience as a Data Engineer writing code to extract, ingest, process and store data within SQL, NoSQL and MPP databases like SnowflakeStrong development experience with Python (or ScalaJava)Experience with complex SQL, batch and streaming pipelines with Apache Spark framework.Knowledge of relational and dimensional data modelingExperience with data quality checks, data validation and data anomaly detection.Experience with workflow management engines like AirflowExperience with Git, CICD pipelines, Docker, KubernetesExperience with architecting solutions on AWS or similar public clouds.Experience with Microservices and Event Driven Architecture
Additional Desired Competencies
Experience with offline and online feature engineering solutions for Machine LearningPrevious domain knowledge in the digital marketingExperience with API development to expose data productsExperience with building Data and ML platforminfrastructure
Perks
Competitive salary.Competitive annual bonus targets.401k with dollar-for-dollar match, up to 6% of eligible earnings (base, bonus). Plus, additional company contribution.RSU grants (Long Term Incentives) for approved roles.Comprehensive medical, dental, vision and life insurance.17 paid holidays per year, including 3 floating holidays.Annual Paid Time Off (PTO), with separate sick daysGenerous discounts on SkinMedica skin care products.Discounted aesthetic treatment days multiple times a year.$600 worth of Alle benefits each year to use towards aesthetic treatments and products.Eligible for donation matching to over 1.5 million nonprofit organizations.
Applicable only to applicants applying to a position in any location with pay disclosure requirements under state or local law:
The compensation range described below is the range of possible base pay compensation that the Company believes in good faith it will pay for this role at the time of this posting based on the job grade for this position. Individual compensation paid within this range will depend on many factors including geographic location, and we may ultimately pay more or less than the posted range. This range may be modified in the future.We offer a comprehensive package of benefits including paid time off. (vacation, holidays, sick), medicaldentalvision insurance and 401(k) to eligible employees.This job is eligible to participate in our short-term incentive programs.This job is eligible to participate in our long-term incentive programs.
Note: No amount of pay is considered to be wages or compensation until such amount is earned, vested and determinable. The amount and availability of any bonus, commission, incentive, benefits, or any other form of compensation and benefits that are allocable to a particular employee remains in the Company's sole and absolute discretion unless and until paid and may be modified at the Company's sole and absolute discretion, consistent with applicable law.

Compensation Range (Minimum - Maximum)

$109,500$208,000 USD

Core Values
Be Humble: You're smart yet always interested in learning from others.Work Transparently: You always deal in an honest, direct and transparent way.Take Ownership: You embrace responsibility and find joy in having the answers.Learn More: Through blog posts, newsletters, podcasts, video tutorials and meetups you regularly self educate and improve your skill set.Show Gratitude: You show appreciation and return kindness to those you work with.
Perks For Our Full Time Employees
Competitive salary.Competitive annual bonus targets.401k with dollar for dollar match, up to 6% of eligible earnings (base, bonus). Plus additional company contribution.RSU grants (Long Term Incentives) for approved roles.Comprehensive medical, dental, vision and life insurance.17 paid holidays per year, including 3 floating holidays.Annual Paid Time Off (PTO), with separate sick days12 weeks paid Parental LeaveCaregiver LeaveAdoption and Surrogacy Assistance PlanFlexible workplace accommodations.We celebrate our wins with opportunities to attend Lakers, Knicks, Anaheim Ducks, Anaheim Angels and NY Rangers games.Opportunities to attend concerts, festivals and other live entertainment events in recognition of delivering great work.Tuition reimbursement.Attend a tech or marketing conference of your choice each year.A MacBook Pro and accompanying hardware to do great work.A modern productivity toolset to get work done: Slack, Miro, Loom, Lucid, Google Docs, Atlassian and more.Generous discounts on SkinMedica skin care products.Discounted aesthetic treatment days multiple times a year.$600 worth of Alle benefits each year to use towards aesthetic treatments and products.Eligible for donation matching to over 1.5 million nonprofit organizations.Attend AWS Re:Invent in person (Las Vegas) or virtually each year (for certain roles)>> the resume: <<Professional SummaryOver 8+ years of progressive professional experience in analysis, Design, Development and Implementation as a Sr. Data Engineer and Data Modeler/Data Analyst.Strong experience in Data Migration, Data Cleansing, Transformation, Integration, Data Import, and Data Export.Good knowledge in Data Quality & Data Governance practices & processes.Highly Qualified with about 5 years of experience in building Big data applications, creating data lakes to manage structured and unstructured data.Well versed with Agile with Scrum, Waterfall Model and Test-driven Development (TDD) methodologies.Server Reporting Services (SSRS) and SQL Server Integration Services (SSIS).Worked with Java based ETL tool, Talend and Implemented Integration solutions for cloud platforms with Informatica Cloud.Extensive experience in setting up CI CD pipelines using tools such as Jenkins, Github Nexus, and Maven.Experience developing On - premise and Real Time processes.Experience on Migrating SQL database to Azure data lake, Azure data lake analytics, Azure SQL database, Data Bricks and Azure SQL Data Warehouse and Controlling and granting database access and migrating on premise databases to Azure Data Lake Store using Azure Data FactoryStrong experience in Big Data technologies like Spark, Spark SQL, Pyspark, Hadoop, HDFS, Hive, Sqoop, Flume, Kafka, Spark StreamingExperience in job workflow scheduling and monitoring tools like Oozie, M2.Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.Experience in designing dashboards, reports, performing ad-hoc analysis and visualizations using Tableau, Power BI.Hands-on experience on NoSQL databases like Snowflake, HBase, Cassandra and Mongo DB.Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in GCP and coordinating tasks among the team.Expertise in DBMS conceptsExperience in GCP Dataproc, GCS, Cloud functions, Big Table and Big Query.Experience in writing and executing unit, system, integration and UATscripts in data warehouse projects.Worked on different file formats like Parquet, Avro, ORC and Flat files.Designing star schema, Snowflake schema for Data Warehouse, ODS architectureExperience with Apache Spark ecosystem using Spark-Core, SQL, Data Frames and RDD's.Experienced in data manipulation using python.Hands-on experience working at Amazon Web Services (AWS) using Elastic Map Reduce Redshift, and EC2 for data processing, AWS glue, RDS, S3, Lambda.Proficient in installing, configuring and using Apache Hadoop ecosystems such as MapReduce, Hive, Pig, Flume, Yarn, HBase, Sqoop, Spark, Storm, Kafka, Oozie, and Zookeeper.Experience in integrating Kafka with Spark streaming for high speed data processing.Extensive experience in developing and driving strategic direction of SAP operating system (SAP ECC) and SAP business intelligence (SAP BI) systemDevelop effective working relationships with client teams to understand and support requirements, develop tactical and strategic plans to implement technology solutions, and effectively manage client expectations.An excellent team member with an ability to perform individually, good interpersonal relations, strong communication skills, hardworking and high level of motivation.
SkillsTechnical SkillsHadoop/Spark Ecosystem:Hadoop, MapReduce, Hive/impala, YARN, Kafka, Flume, Sqoop, Oozie, Zookeeper, SparkDatabases: Oracle, My Sql, SQL Server, PostgreSQL, HBase, Snowflake, Cassandra, Mongo DBCloud computing : Amazon Web Services (AWS), Amazon Redshift, MS Azure, Azure blob storage, Azure Data Factory, Azure Synapse & Google cloud Platform(Big Query, Big Table, Dataproc)BI Tools: Business Objects XI, Tableau 9.1, Power BIQuery Languages: SQL, PL/SQL, T-SQLScripting Languages: Unix, PythonOperating Systems: Linux, Windows, Ubuntu, UnixSDLC Methodology: Agile, Scrum, Waterfall, UML
Work History10/2020toCurrentSr. Data EngineerHumana Inc.–Riverton,WY,Involved in requirements gathering, analysis, design, development, change management, deploymentInvolved in development of real time streaming applications using PySpark, Apache Flink, Kafka, Hive on distributed Hadoop ClusterUtilized Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine Learning use cases under Spark ML and MilibExtracted data from heterogeneous sources and performed complex business logic on network data to normalize raw data which can be utilized by BI teams to detect anomaliesDesigned and developed Flink pipelines to consume streaming data from Kafka and applied business logic to massage and transform and serialize raw dataDeveloped a common Flink module for serializing and deserializing AVRO data by applying schemaDeveloped Spark streaming pipeline to batch real time data, detect anomalies by applying business logic and write anomalies to Hbase tableImplemented layered architecture for Hadoop to modularize designDeveloped framework scripts to enable quick developmentDesigned reusable shell scripts for Hive, Sqoop, Flink and PIG jobsStandardize error handling, logging and metadata management processesIndexed processed data and created dashboards and alerts in splunk to be utilized/ action by support teamsResponsible for operations and support of Big data Analytics platform, Splunk and Tableau visualizationManaged, developed, and designed dashboard control panel for customers and Administrators using Tableau, PostgresQL and REST API callsDesigned and Developed applications using Apache Spark, Scala, Python, Nifi, S3, AWS EMR on AWS cloud to format, cleanse, validate, create schema and build data stores on S3Developed CI-CD pipelines to automate build and deploy to Dev, QA, and production environmentsSupported production jobs and developed several automated processes to handle errors and notificationsAlso, tuned performance of slow jobs by improving design and configuration changes of PySpark jobsCreated standard report Subscriptions and Data Driven Report SubscriptionsTools: Hadoop, Map Reduce, Spark, Spark MLLib, Tableau, SQL, Excel, PIG, Hive, AWS, PostgresQL, Python, PySpark, Flink, SQL Server 2012, T-SQL, CI-CD, Git, XML, Tableau.01/2018to09/2020Sr. GCP Data EngineerCognizant Technology Solutions–Lake Forest,CA,Developed multi cloud strategies in better using GCP (for its PAAS) and Azure (for its SAAS)Involved in loading and transforming large sets of the structured, semi-structured dataset and analysing them by running Hive queriesDeveloped custom python program including CI/CD rules for Google cloud data catalog for metadata managementDesign and develop spark job with Scala to implement end to end data pipeline for batch processingDo fact dimensional modeling and proposed solution to load itProcessing data with Scala, spark, spark SQL and load in hive partition tables in parquet file formatDevelop spark job with partitioned RDD (like hash, range, custom) for faster processingDevelop and deploy the outcome using spark and Scala code in Hadoop cluster running on GCPDevelop near real time data pipeline using flume, Kafka and spark stream to ingest client data from their web log server and apply transformationPerforms data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using ERWIN and MB MDRAssist service developers in finding relevant content in the existing reference modelsLike Access, Excel, CV, Oracle, flat files using connectors, tasks and transformations provided by AWS Data PipelineUtilized Spark SQL API in PySpark to extract and load data and perform SQL queriesWorked on developing Pyspark script to encrypting the raw data by using Hashing algorithms concepts on client specified columnsResponsible for Design, Development, and testing of the database and Developed Stored Procedures, Views, and TriggersDeveloped Python-based API (RESTful Web Service) to track revenue and perform revenue analysisCompiling and validating data from all departments and Presenting to Director OperationDevelop SQOOP script and SQOOP job to ingest data from client provided database in batch fashion on incremental basisUse DISTCP to load files from S3 to HDFS and Processing, cleansing and filtering data using Scala, Spark, Spark SQL, HIVE, Impala Query and Load in Hive tables for data scientists to apply their ML algorithms and generate recommendations as part of data lake processing layerBuild data pipelines in airflow in CP for ETL related jobs using different airflow operators both old and newer operatorsCreated Big Query authorized views for row level security or exposing the data to other teamsGood knowledge in using cloud shell for various tasks and deploying services.06/2016to12/2017Big Data EngineerHoneywell–Altamonte Springs,FL,Implemented a generic ETL framework with high availability for bringing related data for Hadoop & Cassandra from various FRANCHI sources using sparkExperienced in using Platfora a data visualization tool specific for Hadoop, and created various Lens and Viz boards for a real-time visualization from hive tablesQueried and analysed data from Cassandra for quick searching, sorting and grouping through COLImplemented various Data Modelling techniques for CassandraJoined various tables in Cassandra using spark and Scala and ran analytics on top of themParticipated in various upgrades and troubleshooting activities across enterprisesKnowledge in performance troubleshooting and tuning Hadoop clustersApplied Spark advanced procedures like text analytics and processing using in-memory processingImplemented Apache Drill on Hadoop to join data from SQL and No SQL databases and store it in HadoopCreated architecture stack blueprint for data access with NoSQL Database Cassandra;Brought data from various sources into Hadoop and Cassandra using KafkaExperienced in using Tidal enterprise scheduler and Ooze Operational Services for coordinating the cluster and scheduling workflowsApplied spark streaming for real time data transformingCreated multiple dashboards in tableau for multiple business needsInstalled and configured Hive and written Hive UDFs and used piggy bank, a repository of UDF's for Pig LatinImplemented Partitioning, Dynamic Partitions and Buckets in HIVE for efficient data accessExported the analysed data to the relational databases using Sqoop for visualization and to generate reports for the BI team Using TableauImplemented Composite server for the data virtualization needs and created multiple views for restricted data access using a REST APIDevised and led the implementation of next generation architecture for more efficient data ingestion and processingCreated and implemented various shell scripts for automating the jobsImplemented Apache Sentry to restrict the access on the hive tables on a group levelEmployed AVRO format for the entire data ingestion forecaster operation and less space utilizationExperienced in managing and reviewing Hadoop log filesWorked in an Agile environment, and used rally tools to maintain the user stories and tasksWorked with Enterprise data support teams to install Hadoop updates, patches, version upgrades as required and fixed problems, which were raised after the upgradesImplemented test scripts to support test-driven development and continuous integrationUsed Spark for Parallel data processing and better performancesTools: Map Reduce, HDFS, Hive, pig, Impala, Cassandra, spark, Scala, solr, Java, SQL, Tableau, PIG, Zookeeper, Sqoop, Teradata, CentOS, Pentaho.06/2014to11/2015Data EngineerITC InfoTech–City,,IndiaWorked on configuration and monitoring Hadoop cluster using Cloudera distributionInvolved in migrating data from on prem Cloudera cluster to AWS EC2 instances deployed on EMR cluster and developed ETL pipeline to extract logs and store in AWS S3 Data Lake and further processed it using PySparkMoved files between HDFS and AWS S3 and worked with S3 bucket in AWS on a regular basisResponsible for developing data pipelines using Flume, Sqoop and Pig to extract the data from weblogs and store them in HDFSMigrated data between various data sources like Teradata, Oracle, and MySQL to HDFS by using SqoopUsed HCatalog to access Hive table metadata from MapReduce and Pig codeDeveloped a data pipeline using Kafka and Storm for streaming data and to store it into HDFSUsed Informatica Powercenter for cleaning, managing, and integrating data from different sources for ETL and loaded into a single warehouse repositoryUsed Impala to read, write and query the Hadoop data in HDFS from HBase and constructed Impala scripts to reduce query response timeAnalysed data stored in S3 buckets using SQL, PySpark and stored the processes data in Redshift and validated data sets by implementing Spark componentsPerformed ETL operations using Python, SQL on many data sets to obtain metricsPrepared data according to analyst requirements on the extracted data using Pandas and NumPy modules in PythonInvolved in designing and developing automation test scripts using PythonInvolved in writing multiple python scripts to extract data from different API’sCreated HBase tables using Shell to load large sets of data from different databasesInvolved in scheduling Time based Oozie workflow engine to run multiple Hive and Pig jobsDeveloped flow XML files using Apache NiFi to process and ingest data into HDFSWorked on performance tuning of Apache NiFi workflow to optimize the data ingestion speedsResponsible for collecting and aggregating large amounts of log data using Flume and staging it into HDFS for further analysisWorked on integration of Apache Storm with Kafka to perform web analytics and upload streaming data from Kafka to HBase and HiveResponsible for developing data pipelines using Apache Kafka by implementing Kafka producers and consumersUsed Hive optimization techniques like partitioning and bucketing to provide better performance with HiveQL queriesLoaded large amounts of data to HBase using MapReduce jobsWorked on developing UDFs to work with Hive and wrote our tests in ScalaUsed zookeeper to maintain configurations across clusters and for better synchronization, grouping and reliable distributed coordinationWorked with Kerberos and Apache sentry for security and authorization on HadoopUsed Git for version controlTools: Cloudera CDH4, Hadoop 2.x (HDFS, MapReduce, Yarn, Sqoop, Hive), AWS, EC2, S3, Redshift, Impala, Spark, Pig, SQL, HBase, Kafka, Zookeeper, Flume, Oozie, HCatalog, NiFi, Storm, Informatica, Python, MySQL, Scala, Teradata, Oracle, Git.
EducationExpected in2013totoBachelor’s:Computer ScienceCBIT-Hyderabad,GPA:
SkillsTechnical SkillsHadoop/Spark Ecosystem:Hadoop, MapReduce, Hive/impala, YARN, Kafka, Flume, Sqoop, Oozie, Zookeeper, SparkDatabases: Oracle, My Sql, SQL Server, PostgreSQL, HBase, Snowflake, Cassandra, Mongo DBCloud computing : Amazon Web Services (AWS), Amazon Redshift, MS Azure, Azure blob storage, Azure Data Factory, Azure Synapse & Google cloud Platform(Big Query, Big Table, Dataproc)BI Tools: Business Objects XI, Tableau 9.1, Power BIQuery Languages: SQL, PL/SQL, T-SQLScripting Languages: Unix, PythonOperating Systems: Linux, Windows, Ubuntu, UnixSDLC Methodology: Agile, Scrum, Waterfall, UML>>. The result is, 

Fit assessment: Good Fit