Job description:
title: sr. data engineerlocation: austin, txduration: 6 monthspay rate: $65hr - $70hr on w2 requirements:bachelors degree in computer science or related field plus 4+ years of relevant work experience or a master's degree plus 2+ years of relevant work experience or a phd plus 0-1 years of relevant experiencein lieu of a degree, qualified candidates would require 8+ years of relevant professional experienceexcellent with sqlexcellent programming skills in python and ideally demonstrating an aptitude via experience with multiple languagesexcellent understanding of patterns for data ingest into data warehouse, ingest, cleansing, standardizing, etc., in addition to different data structures like normalized, denormalized, starexcellent experience supporting snowflake data warehouse, including snowpipe (including streams), tasks, transformations, views, dynamic tables. this should include advanced skills in ensuring efficient utilization of snowflake compute and the ability to optimize workloads and warehousebroad experience with cloud paas capabilities, ideally aws cloudwatch, lambda, step functions, snssqs, dynamodb, etc.experience utilizing reporting and data insights toolsexperience in supporting analytics teams data needs, in addition to customer and business reporting. what you'll be doing:provides data engineering support for areas such as finance, sales, business intelligence, product development andor other business usersworks with data consumers and project managers to determine logical and physical database designs for analytics modelscreates and maintains optimal data pipeline architecturesensures architectures are aligned with and support business requirements;ensures that required data is available, can be trusted and is readily accessible by those who need itinfluences the data infrastructure roadmapidentifies, designs and implements internal data management process improvementsplays a key role in application development projects to evolve the companys database architecture and designworks with internal and external data providers on data validation, providing feedback and making customized changes to data feeds and data mappings for analytical and operational useautomates manual processes, transforming them into repeatable capabilitiesworks with fellow team members to ensure design, development and execution align with and support adjacencies and adhere to established architectural standards. top 5 must haves:excellent with sqlexcellent programming skills in python and ideally demonstrating an aptitude via experience with multiple languagesexcellent understanding of patterns for data ingest into data warehouse, ingest, cleansing, standardizing, etc., in addition to different data structures like normalized, denormalized, star;excellent experience supporting snowflake data warehouse, including snowpipe (including streams), tasks, transformations, views, dynamic tables. this should include advanced skills in ensuring efficient utilization of snowflake compute and the ability to optimize workloads and warehousebroad experience with cloud paas capabilities, ideally aws cloudwatch, lambda, step functions, snssqs, dynamodb, etc.

Resume:
<<summaryover 3 years of professional it experience indata engineeringanddata analyticsusing various languages and tools likesql, python, big/hadoopextensive experience on data engineering field including ingestion, datalake, datawarehouse, reporting and analytics.strong knowledge and experience on data analysis, data lineage, big data pipelines, data quality, data reconciliation, data transformation rules, data flow diagram including data replication, data integration and data orchestration toolsknowledge and experience on aws services like redshift spectrum,s3,glue,athena, lambda,cloudwatch and emrs like hive, prestoextensively used etl methodology for performing data migration, extraction, transformation and loading usingtalendand designed data conversions from wide variety of source systems.experience d in data ingestion projects to inject data intodata lakeusing multiple sources systems using talend bigdata.good technical skills insql server,etldevelopment usinginformatica tool. /sqlto integrate of complex oltp and olap database models and data marts, worked extensively onoracle, sql server.experience in all the life cycle phases of the projects onlarge data setsand experience with performancetuningandtroubleshooting.ability to work effectively with associates at all levels within the organization.strong background in mathematics and have very good analytical and problem-solving skills. skills big data ecosystems:-hdfs, mapreduce, spark, kafka, hive, airflow, stream sets, hbase, flume, zookeeper, nifi, sentry, ranger.scripting language:-python, powershell scripting, pig latin, hiveql.cloud environment:-amazon web services (aws), microsoft azure,nosql databasedatabase:-mysql, oracle, sql server, postgresql, db2.version control:-git, svn, bitbucket.etl tools:-tableau, microsoft excel, informatica, power bi, r, google data studio. experience data engineer, 01/2022-currenthoneywell– ,worked on architecture design for multistate implementation or deployment.implement one time data migration of multistate level data from sql server to snowflake by using python and snowsql.day to-day responsibility includes developing etl pipelines in and out of data warehouse, develop major regulatory and financial reports using advanced sql queries in snowflake.stage the api or kafka data(in json file format) into snowflake db by flattening the same for different functional services.build docker images to run airflow on local environment to test the ingestion as well as etl pipelines.building/maintaining docker container clusters managed by kubernetes. utilization of kubernetes and docker for the runtime environment of the ci/cd system to build, test and deploy.created airflow dags to schedule the ingestions, etl jobs and various business reports.createdairflowscheduling scripts in python.xcluster capacity planning along with operations team and management team and cluster maintenance as well as creation and removal of nodes,hdfssupport and maintenance.strong knowledge ofrack awarenesstopology in thehadoop cluster.involved in loading data fromlinux file systemtohadoop distributed file system.responsible for building scalable distributed data solutions usinghadoop.experience in managing and reviewinghadoop log files.data migration fromrdmstohadoopusingsqoopfor analysis and implementedooziejobs for automatic data imports from source.createdhbasetables to store various data formats ofpiidata coming from different portfolios.strong in exporting the analyzed and processed data to therelational databasesusingsqoopfor visualization and for generation of reports for the team.data engineer, 01/2021- 12/2021honeywell– ,prepared etl design document which consists of the database structure, change data capture, error handling, restart and refresh strategies.worked with different feeds data like json, csv, xml,dat and implemented data lake concept.developed informatica design mappings using various transformations.designing and building multi-terabyte, full end-to-end data warehouse infrastructure from the ground up on confidentialredshiftfor large scale data handling millions of records every day.optimizing and tuning theredshiftenvironment, enabling queries to perform up to 100x faster for tableau and sas visual analytics.wrote various data normalization jobs for new data ingested intoredshift.advanced knowledge on confidentialredshiftand mpp database concepts.migrated on premise database structure to confidentialredshiftdata warehousedeveloped udf in scala to implement the business logic.developed spark applications in scala on distributed environment to load huge number of json files with different schema in to hive tables.most of the infrastructure is on aws, used,aws emrdistribution for hadoopaws s3for raw file storageaws ec2for kafkausedaws lambdato perform data validation, filtering, other transformations for every data change in a dynamodb table and load the transformed data to another data store.createdairflowscheduling scripts in python.programmedetl functionsbetween oracle and amazon redshift.sql developer, 06/2019- 12/2023ihs markit– ,gathered business requirements and converted them into new t-sql stored procedures in visual studio for database project.performed unit tests on all code and packages.analyzed requirement and impact by participating in joint application development sessions with business client online.performed and automated sql server version upgrades, patch installs and maintained relational databases.performed front line code reviews for other development teams.modified and maintained sql server stored procedures, -hoc queries, and ssis packages used in the search engine optimization process.updated existing and created new reports using microsoft sql server reporting services. team consisted of 2 developers.created files, views, tables and data sets to support sales operations and analytics teamsmonitored and tuned database resources and activities for sql server databases. education master of science:data analytics engineering,expected in 12/2022-george mason university-fairfax,vagpa:status-3.8 gpabachelor of science:electronics & communication engineering,expected in 06/2019-kl university-guntur,gpa:status-.