Job description:
role: lead data engineerlocation: remoteduration: long term 8 to 10 years of experience working in data engineering, business intelligence or related field delivering large scale data platforms and services. ideal candidate will have the following skills: a successful history of manipulating, processing and extracting value from large, disconnected datasets. expertise in data modeling, data access and data storage techniques. the candidate should have foundational knowledge of dimensional modeling to build a data warehouse strong experience building transformational logic using sql on relational sql and nosql databases. skills using dbt framework for data transformation is plus robust programming skills and strong experience with python to ingest, transform and load the data experience building and optimizing big data data pipelines, architectures and data sets using tools like airflow. experience with aws cloud services: ec2, emr, rds, redshift

Resume:
<<summarystrong it experience in aws cloud computing, bigdata & data warehousing.strong business domain knowledge in sales segmentation & territory planning.strong coding & sql experience in python and redshifthaving an experience in agile and deliver the results based on stories and task assigned on the sprint.implementation knowledge in server-less architecture using aws cloud computing.having good hands on experience in git as well as ci/cd pipeline integration.having good hands on experience in building data model based on requirements. skills programming languages: java, python, unix shell scriptingcloud technologies: amazon web services (aws)version control: aws codecommit, git, github repositoriescontinuous integration/continuous delivery(ci/cd): aws codedeploy aws codepipelinedatabases and sql: redshift, oracle, netezza, nosql (ddb & elasticsearch)big data technologies: hadoop, hive, ozzie, airflow, spark, pyspark, scoop, flume experience 01/2018tocurrentsenior data engineersplunk ,collaborated with product owners to gather requirement to help build a solution for sales segmentation.identified key use cases and associated reference architectures for market segments and industry verticals.worked as part of project teams to coordinate database and pipeline development and determine project scopes and limitations.collaborated with solution architects to define database and analytics engagement strategies for operational territories and key accounts.developed and managed enterprise-wide big data environments.specified user access levels for each database segment to protect database integrity and company information.developed and implemented security initiatives to protect important company data.established hardware requirements and devised storage capacity solutions and choose wisely on the aws services by outweigh the requirement and business use-case. 12/2015to 12/2017big data engineernike inc ,perform as a big data developer and work in various phases like data ingestion, data integration and transformation.create a sqoop import command and pull the data (both tables and all tables in db's) from mysql db to hdfs as part of data ingestioncreates a mapreduce programs as well as spark programs to parse, analyze and implement the solutions based on its customer need.perform operations to filter the records from db, boundary query, incremental update or insert in the sqoop import commands.perform operations to insert the data directly to hive tables, change the delimiters and end line character and change the formats like textfile, avrodatafile, orc file into hdfs systems using sqoop import commands.perform sqoop eval function to evaluate the data in mysql.perform sqoop export function to export the data from hdfs and loaded it into mysql and understand the delimiter of the file, define the no of mapper to perform, etc.create a flume config file to ingest the data from various source systems like (spool directory, netcat, exec, sequence, etc.) and loaded it into avro, hdfs etc.define a proper channels(memory, file ,etc.) and loaded all the data comes from source to sinks.used the interceptor to modify the data comes from source like filtering (regex), include the timestamp, search and replace, etc.create a mapreduce program using eclipse ide and execute it through jar file.create a mapper class and using map method generate the key value pair as (node_with_date, system utilization) by writing in context application.create a reducer class and using reduce method, generate the final output of system utilization per node_with_date.implements custom based writable class to get the two values in the mapoutputvalue class.implements custom based partitioner class to assign a appropriate data to the right reducer(basically try to achieve multi-level group by function with mr limitations).implements pattern matching logic for semi-structured data and routed the good and bad records separately.create a custom based input format as well as record reader to parse the xml file.hands-on in creating the broadcast variables.hands-on in sparksql, via class object in scala, convert df to rdd, temptable schema creation and so on. 09/2009to 04/2015etl developerallegis group ,created and implemented complex business intelligence solutions.created conceptual, logical and physical data models for use in different business areas.developed and managed enterprise-wide data analytics environments.identified, protected and leveraged existing data.monitored multiple databases to keep track of all company inventory.assisted maintenance team with completion of preventive maintenance and unscheduled service needs.monitored installations to ensure compliance with local codes and industry best practices. education and trainingexpected intotomaster of science information technologyamity university,noida, india,gpa:expected in 05/2008totobachelor of engineering college of engineering guindy, anna university,chennai, india,gpa:.