Job description:
the senior manager, data architecture & data engineering is responsible for translating business strategic goals into future-state data engineering and data science solutions. this includes delivery full life-cycle from execution roadmaps, to implementation, and to production operations. this role drives the creation and maintenance of enterprise data and analytics capabilities, data reference architectures, and integrated data & analytics technology roadmaps that will guide our strategic investments and enterprise strategy.translate business strategic goals into actionable future-state data engineering and data science architectures and solutions.delivery of full life-cycle data solutions from execution roadmaps, to implementation, and to operations. this includes a specific focus on elt and etl data pipeline solutions from inception to production.lead end to end solution delivery of enterprise data warehouse, data marts, and data lake with governance.drive the creation and maintenance of enterprise data and analytics capabilities, data solutions, and integrated data & analytics technology roadmaps that will guide our strategic investments and enterprise strategy.deliver architecture connecting data science, data engineering, and data analytics practices together to deliver impactful data products.strong partnership with devops, dataops, business partners, engineering teams, and data management.a minimum of ten (10) years of experience in it with a minimum (6) years in data architecture and data solutions are required.successful track-record in managing a team of data engineers and data scientistsskilled in hybrid cloud data platforms and analytical technologies including azure andor gcp data infrastructure, spark, data lake, data warehouse and data marts, bi reporting with powerbi, metadata management, data governance, master data management, data catalog, and business glossary.ability to document, communicate, implement, and operationalize complex architectures.experience with elt and etl data pipeline solutions from inception to production.proficient in sql, nosql and columnar databases.experience with sparkpyspark and json.excellent grasp of computer science fundamentals - data structures and algorithms.must have panda data frames knowledge and best practices, execution engines, dags (airflow, dagster, luigi, etc), and kubernetes.proficient with gitlabfamiliarity with azure data factory preferred, google data flow or aws glueexperience with biomedical informatics, s3azure blob store, machine learning and ai, unstructured data, and data democratization is preferred.experience with agile methodologies, kanban preferred, and using tools like jira and confluence to facilitate the work.bachelor's degree in computer science or a related discipline required. master's degree preferred. recruiters notes: the role will definitely be hands-on and will require excellent communication skills. the hiring manager mentioned the following technologies: aimachine learning; data warehouses; product support; cloud experienceazure; must be open to new technologies. working with the product development team. act as the point of contact; will create a group that helps the product teams. create data warehouses for analytics. must have excellent communication skills verbal and written. this person will envision strategy and must be able to effectively communicate that strategy to others. the organization has asked me not to advertise the compensation. if you contact me i would be more than willing to share it with you. for over 50 years this non-profit organization has made it their mission to protect patients from unsafe and ineffective medical technologies and practices. more than 5,000 healthcare institutions and systems worldwide including four out of every five u.s. hospitals rely on the services of this organization to guide their operational and strategic decisions. in addition, they serve public and private payers, federal and state agencies, policymakers, ministries of health, associations, and accrediting agencies. benefit from a healthy work-life balance while staying on the leading edge of technology and thriving in an innovative startup-like culture minus the risk. sleep well knowing you are helping achieve a world where safe, high-quality healthcare is accessible to everyone. excellent benefits to include work from home, flex-time and 19 days starting vacation. its a nice place to work!

Resume:
<<professional summaryover 8+ years of progressive professional experience in analysis, design, development and implementation as a sr. data engineer and data modeler/data analyst.strong experience in data migration, data cleansing, transformation, integration, data import, and data export.good knowledge in data quality & data governance practices & processes.highly qualified with about 5 years of experience in building big data applications, creating data lakes to manage structured and unstructured data.well versed with agile with scrum, waterfall model and test-driven development (tdd) methodologies.server reporting services (ssrs) and sql server integration services (ssis).worked with java based etl tool, talend and implemented integration solutions for cloud platforms with informatica cloud.extensive experience in setting up ci cd pipelines using tools such as jenkins, github nexus, and maven.experience developing on - premise and real time processes.experience on migrating sql database to azure data lake, azure data lake analytics, azure sql database, data bricks and azure sql data warehouse and controlling and granting database access and migrating on premise databases to azure data lake store using azure data factorystrong experience in big data technologies like spark sql, pyspark, hadoop, hdfs, hive, sqoop, flume, kafka, spark streamingexperience in job workflow scheduling and monitoring tools like oozie, m2.experience in importing and exporting data using sqoop from hdfs to relational database systems and vice-versa.experience in designing dashboards, reports, performing ad-hoc analysis and visualizations using tableau, power bi.hands-on experience on nosql databases like snowflake, hbase, cassandra and mongo db.experience in building and architecting multiple data pipelines, end to end etl and elt process for data ingestion and transformation in gcp and coordinating tasks among the team.expertise in dbms conceptsexperience in gcp dataproc, gcs, cloud functions, big table and big query.experience in writing and executing unit, system, integration and uatscripts in data warehouse projects.worked on different file formats like parquet, avro, orc and flat files.designing star schema, snowflake schema for data warehouse, ods architectureexperience with apache spark ecosystem using spark-core, sql, data frames and rdd's.experience d in data manipulation using python.hands-on experience working at amazon web services (aws) using elastic map reduce redshift, and ec2 for data processing, aws glue, rds, s3, lambda.proficient in installing, configuring and using apache hadoop ecosystems such as mapreduce, hive, pig, flume, yarn, hbase, sqoop, spark, storm, kafka, oozie, and zookeeper.experience in integrating kafka with spark streaming for high speed data processing.extensive experience in developing and driving strategic direction of sap operating system (sap ecc) and sap business intelligence (sap bi) systemdevelop effective working relationships with client teams to understand and support requirements, develop tactical and strategic plans to implement technology solutions, and effectively manage client expectations.an excellent team member with an ability to perform individually, good interpersonal relations, strong communication skills, hardworking and high level of motivation. skills technical skills hadoop/spark ecosystem:hadoop, mapreduce, hive/impala, yarn, kafka, flume, sqoop, oozie, zookeeper, sparkdatabases: sql server, postgresql, hbase, snowflake, cassandra, mongo dbcloud computing : amazon web services (aws), azure blob storage, azure data factory, azure synapse & google cloud platform(big query, big table, dataproc)bi tools: business objects xi, tableau 9.1, power biquery languages: /sql, t-sqlscripting languages: unix, pythonoperating systems: linux, windows, ubuntu, unixsdlc methodology: agile, scrum, waterfall, uml work history 10/2020tocurrentsr. data engineerhumana inc.– ,involved in requirements gathering, analysis, design, development, change management, deploymentinvolved in development of real time streaming applications using pyspark, apache flink, kafka, hive on distributed hadoop clusterutilized apache spark with python to develop and execute big data analytics and machine learning applications, executed machine learning use cases under spark ml and milibextracted data from heterogeneous sources and performed complex business logic on network data to normalize raw data which can be utilized by bi teams to detect anomaliesdesigned and developed flink pipelines to consume streaming data from kafka and applied business logic to massage and transform and serialize raw datadeveloped a common flink module for serializing and deserializing avro data by applying schemadeveloped spark streaming pipeline to batch real time data, detect anomalies by applying business logic and write anomalies to hbase tableimplemented layered architecture for hadoop to modularize designdeveloped framework scripts to enable quick developmentdesigned reusable shell scripts for hive, sqoop, flink and pig jobsstandardize error handling, logging and metadata management processesindexed processed data and created dashboards and alerts in splunk to be utilized/ action by support teamsresponsible for operations and support of big data analytics platform, splunk and tableau visualizationmanaged, developed, and designed dashboard control panel for customers and administrators using tableau, postgresql and rest api callsdesigned and developed applications using apache spark, scala, python, nifi, s3, aws emr on aws cloud to format, cleanse, validate, create schema and build data stores on s3developed ci- , and production environmentssupported production jobs and developed several automated processes to handle errors and notificationsalso, tuned performance of slow jobs by improving design and configuration changes of pyspark jobscreated standard report subscriptions and data driven report subscriptionstools: hadoop, map reduce, spark mllib, tableau, sql, excel, pig, hive, aws, postgresql, python, pyspark, flink, sql server 2012, t- -cd, git, xml, tableau. 01/2018to 09/2020sr. gcp data engineercognizant technology solutions– ,developed multi cloud strategies in better using gcp (for its paas) and azure (for its saas)involved in loading and transforming large sets of the structured, semi-structured dataset and analysing them by running hive queriesdeveloped custom python program including ci/cd rules for google cloud data catalog for metadata managementdesign and develop spark job with scala to implement end to end data pipeline for batch processingdo fact dimensional modeling and proposed solution to load itprocessing data with scala, spark sql and load in hive partition tables in parquet file formatdevelop spark job with partitioned rdd (like hash, range, custom) for faster processingdevelop and deploy the outcome using spark and scala code in hadoop cluster running on gcpdevelop near real time data pipeline using flume, kafka and spark stream to ingest client data from their web log server and apply transformationperforms data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using erwin and mb mdrassist service developers in finding relevant content in the existing reference modelslike access, , oracle, flat files using connectors, tasks and transformations provided by aws data pipelineutilized spark sql api in pyspark to extract and load data and perform sql queriesworked on developing pyspark script to encrypting the raw data by using hashing algorithms concepts on client specified columnsresponsible for design, development, and testing of the database and developed stored procedures, views, and triggersdeveloped python-based api (restful web service) to track revenue and perform revenue analysiscompiling and validating data from all departments and presenting to director operationdevelop sqoop script and sqoop job to ingest data from client provided database in batch fashion on incremental basisuse distcp to load files from s3 to hdfs and processing, cleansing and filtering data using scala, spark sql, hive, impala query and load in hive tables for data scientists to apply their ml algorithms and generate recommendations as part of data lake processing layerbuild data pipelines in airflow in cp for etl related jobs using different airflow operators both old and newer operatorscreated big query authorized views for row level security or exposing the data to other teamsgood knowledge in using cloud shell for various tasks and deploying services. 06/2016to 12/2017big data engineerhoneywell– ,implemented a generic etl framework with high availability for bringing related data for hadoop & cassandra from various franchi sources using sparkexperience d in using platfora a data visualization tool specific for hadoop, and created various lens and viz boards for a real-time visualization from hive tablesqueried and analysed data from cassandra for quick searching, sorting and grouping through colimplemented various data modelling techniques for cassandrajoined various tables in cassandra using spark and scala and ran analytics on top of themparticipated in various upgrades and troubleshooting activities across enterprisesknowledge in performance troubleshooting and tuning hadoop clustersapplied spark advanced procedures like text analytics and processing using in-memory processingimplemented apache drill on hadoop to join data from sql and no sql databases and store it in hadoopcreated architecture stack blueprint for data access with nosql database cassandra;brought data from various sources into hadoop and cassandra using kafkaexperience d in using tidal enterprise scheduler and ooze operational services for coordinating the cluster and scheduling workflowsapplied spark streaming for real time data transformingcreated multiple dashboards in tableau for multiple business needsinstalled and configured hive and written hive udfs and used piggy bank, a repository of udf's for pig latinimplemented partitioning, dynamic partitions and buckets in hive for efficient data accessexported the analysed data to the relational databases using sqoop for visualization and to generate reports for the bi team using tableauimplemented composite server for the data virtualization needs and created multiple views for restricted data access using a rest apidevised and led the implementation of next generation architecture for more efficient data ingestion and processingcreated and implemented various shell scripts for automating the jobsimplemented apache sentry to restrict the access on the hive tables on a group levelemployed avro format for the entire data ingestion forecaster operation and less space utilizationexperience d in managing and reviewing hadoop log filesworked in an agile environment, and used rally tools to maintain the user stories and tasksworked with enterprise data support teams to install hadoop updates, patches, version upgrades as required and fixed problems, which were raised after the upgradesimplemented test scripts to support test-driven development and continuous integrationused spark for parallel data processing and better performancestools: map reduce, hdfs, hive, pig, impala, cassandra, spark, scala, solr, java, sql, tableau, pig, zookeeper, sqoop, teradata, centos, pentaho. 06/2014to 11/2015data engineeritc infotech–city.indiaworked on configuration and monitoring hadoop cluster using cloudera distributioninvolved in migrating data from on prem cloudera cluster to aws ec2 instances deployed on emr cluster and developed etl pipeline to extract logs and store in aws s3 data lake and further processed it using pysparkmoved files between hdfs and aws s3 and worked with s3 bucket in aws on a regular basisresponsible for developing data pipelines using flume, sqoop and pig to extract the data from weblogs and store them in hdfsmigrated data between various data sources like teradata, oracle, and mysql to hdfs by using sqoopused hcatalog to access hive table metadata from mapreduce and pig codedeveloped a data pipeline using kafka and storm for streaming data and to store it into hdfsused informatica powercenter for cleaning, managing, and integrating data from different sources for etl and loaded into a single warehouse repositoryused impala to read, write and query the hadoop data in hdfs from hbase and constructed impala scripts to reduce query response timeanalysed data stored in s3 buckets using sql, pyspark and stored the processes data in redshift and validated data sets by implementing spark componentsperformed etl operations using python, sql on many data sets to obtain metricsprepared data according to analyst requirements on the extracted data using pandas and numpy modules in pythoninvolved in designing and developing automation test scripts using pythoninvolved in writing multiple python scripts to extract data from different api’screated hbase tables using shell to load large sets of data from different databasesinvolved in scheduling time based oozie workflow engine to run multiple hive and pig jobsdeveloped flow xml files using apache nifi to process and ingest data into hdfsworked on performance tuning of apache nifi workflow to optimize the data ingestion speedsresponsible for collecting and aggregating large amounts of log data using flume and staging it into hdfs for further analysisworked on integration of apache storm with kafka to perform web analytics and upload streaming data from kafka to hbase and hiveresponsible for developing data pipelines using apache kafka by implementing kafka producers and consumersused hive optimization techniques like partitioning and bucketing to provide better performance with hiveql queriesloaded large amounts of data to hbase using mapreduce jobsworked on developing udfs to work with hive and wrote our tests in scalaused zookeeper to maintain configurations across clusters and for better synchronization, grouping and reliable distributed coordinationworked with kerberos and apache sentry for security and authorization on hadoopused git for version controltools: cloudera cdh4, hadoop 2.x (hdfs, mapreduce, yarn, sqoop, hive), aws, ec2, s3, redshift, impala, spark, pig, sql, hbase, kafka, zookeeper, flume, oozie, hcatalog, nifi, storm, informatica, python, mysql, scala, teradata, oracle, git. education expected in2013totobachelor’s:computer sciencecbit-hyderabad,gpa: skills technical skills hadoop/spark ecosystem:hadoop, mapreduce, hive/impala, yarn, kafka, flume, sqoop, oozie, zookeeper, sparkdatabases: sql server, postgresql, hbase, snowflake, cassandra, mongo dbcloud computing : amazon web services (aws), azure blob storage, azure data factory, azure synapse & google cloud platform(big query, big table, dataproc)bi tools: business objects xi, tableau 9.1, power biquery languages: /sql, t-sqlscripting languages: unix, pythonoperating systems: linux, windows, ubuntu, unixsdlc methodology: agile, scrum, waterfall, uml.