Job description:
as a solution architect - data engineering, you will work for an american multinational technology company focusing on various technological domains such as artificial intelligence, online advertising, search engine technology, cloud computing, computer software, quantum computing, e-commerce, and consumer electronics. the job requires expertise in databricks, a cloud-based data engineering platform. key responsibilities:provide technical leadership in a post-sales capacity to guide strategic customers as they design and implement big data projects, ranging from architectural design to data engineering to model deployment.identify and drive new initiatives that enable customers to turn their data into actionable business value and align these initiatives with their business outcomes for continued success.architect production-level workloads, including end-to-end pipeline load performance testing and optimization.deliver tutorials and training to drive community adoption (including hackathons, conference presentations, etc.)work closely with engineering and product management to drive the direction of the databricks product.contribute to databricks sme community. required skillshands-on and technical expertise with apache sparkproven experience of design and implementation experience in big data technologies including hadoop, nosql, mpp, oltp, olap5+ years experience working as either: software engineerdata engineer: query tuning, performance tuning, troubleshooting, and debugging spark andor other big data solutions.comfortable programming in python, scala, or javaexperience using and designing solutions on cloud infrastructure and services, such as aws, azure, or gcpexperience with development tools for cicd, unit and integration testing, automation and orchestration, rest api, bi tools, and sql interfaces. e.g. jenkins.excellent communication skills and comfortable presenting to strategic stakeholdersleadership.experience in customer-facing pre-sales, post-sales, technical architecture guidance, or consulting.passionate about learning new technologies and making customers successfuldata scienceml engineer experience in model selection, model lifecycle, hyper-parameter tuning, model serving, and deep learning, using tools like mlflow desired. life at capgeminicapgemini supports all aspects of your well-being throughout the changing stages of your life and career. for eligible employees, we offer:flexible workhealthcare, including dental, vision, mental health, and well-being programsfinancial well-being programs such as 401(k) and employee share ownership planpaid time off and paid holidayspaid parental leavefamily-building benefits like adoption assistance, surrogacy, and cryopreservationsocial well-being benefits like subsidized backup childelder care and tutoringmentoring, coaching, and learning programsemployee resource groupsdisaster relief

Resume:
<<professional summaryover 8+ years of progressive professional experience in analysis, design, development and implementation as a sr. data engineer and data modeler/data analyst.strong experience in data migration, data cleansing, transformation, integration, data import, and data export.good knowledge in data quality & data governance practices & processes.highly qualified with about 5 years of experience in building big data applications, creating data lakes to manage structured and unstructured data.well versed with agile with scrum, waterfall model and test-driven development (tdd) methodologies.server reporting services (ssrs) and sql server integration services (ssis).worked with java based etl tool, talend and implemented integration solutions for cloud platforms with informatica cloud.extensive experience in setting up ci cd pipelines using tools such as jenkins, github nexus, and maven.experience developing on - premise and real time processes.experience on migrating sql database to azure data lake, azure data lake analytics, azure sql database, data bricks and azure sql data warehouse and controlling and granting database access and migrating on premise databases to azure data lake store using azure data factorystrong experience in big data technologies like spark sql, pyspark, hadoop, hdfs, hive, sqoop, flume, kafka, spark streamingexperience in job workflow scheduling and monitoring tools like oozie, m2.experience in importing and exporting data using sqoop from hdfs to relational database systems and vice-versa.experience in designing dashboards, reports, performing ad-hoc analysis and visualizations using tableau, power bi.hands-on experience on nosql databases like snowflake, hbase, cassandra and mongo db.experience in building and architecting multiple data pipelines, end to end etl and elt process for data ingestion and transformation in gcp and coordinating tasks among the team.expertise in dbms conceptsexperience in gcp dataproc, gcs, cloud functions, big table and big query.experience in writing and executing unit, system, integration and uatscripts in data warehouse projects.worked on different file formats like parquet, avro, orc and flat files.designing star schema, snowflake schema for data warehouse, ods architectureexperience with apache spark ecosystem using spark-core, sql, data frames and rdd's.experience d in data manipulation using python.hands-on experience working at amazon web services (aws) using elastic map reduce redshift, and ec2 for data processing, aws glue, rds, s3, lambda.proficient in installing, configuring and using apache hadoop ecosystems such as mapreduce, hive, pig, flume, yarn, hbase, sqoop, spark, storm, kafka, oozie, and zookeeper.experience in integrating kafka with spark streaming for high speed data processing.extensive experience in developing and driving strategic direction of sap operating system (sap ecc) and sap business intelligence (sap bi) systemdevelop effective working relationships with client teams to understand and support requirements, develop tactical and strategic plans to implement technology solutions, and effectively manage client expectations.an excellent team member with an ability to perform individually, good interpersonal relations, strong communication skills, hardworking and high level of motivation. skills technical skills hadoop/spark ecosystem:hadoop, mapreduce, hive/impala, yarn, kafka, flume, sqoop, oozie, zookeeper, sparkdatabases: sql server, postgresql, hbase, snowflake, cassandra, mongo dbcloud computing : amazon web services (aws), azure blob storage, azure data factory, azure synapse & google cloud platform(big query, big table, dataproc)bi tools: business objects xi, tableau 9.1, power biquery languages: /sql, t-sqlscripting languages: unix, pythonoperating systems: linux, windows, ubuntu, unixsdlc methodology: agile, scrum, waterfall, uml work history 10/2020tocurrentsr. data engineerhumana inc.– ,involved in requirements gathering, analysis, design, development, change management, deploymentinvolved in development of real time streaming applications using pyspark, apache flink, kafka, hive on distributed hadoop clusterutilized apache spark with python to develop and execute big data analytics and machine learning applications, executed machine learning use cases under spark ml and milibextracted data from heterogeneous sources and performed complex business logic on network data to normalize raw data which can be utilized by bi teams to detect anomaliesdesigned and developed flink pipelines to consume streaming data from kafka and applied business logic to massage and transform and serialize raw datadeveloped a common flink module for serializing and deserializing avro data by applying schemadeveloped spark streaming pipeline to batch real time data, detect anomalies by applying business logic and write anomalies to hbase tableimplemented layered architecture for hadoop to modularize designdeveloped framework scripts to enable quick developmentdesigned reusable shell scripts for hive, sqoop, flink and pig jobsstandardize error handling, logging and metadata management processesindexed processed data and created dashboards and alerts in splunk to be utilized/ action by support teamsresponsible for operations and support of big data analytics platform, splunk and tableau visualizationmanaged, developed, and designed dashboard control panel for customers and administrators using tableau, postgresql and rest api callsdesigned and developed applications using apache spark, scala, python, nifi, s3, aws emr on aws cloud to format, cleanse, validate, create schema and build data stores on s3developed ci- , and production environmentssupported production jobs and developed several automated processes to handle errors and notificationsalso, tuned performance of slow jobs by improving design and configuration changes of pyspark jobscreated standard report subscriptions and data driven report subscriptionstools: hadoop, map reduce, spark mllib, tableau, sql, excel, pig, hive, aws, postgresql, python, pyspark, flink, sql server 2012, t- -cd, git, xml, tableau. 01/2018to 09/2020sr. gcp data engineercognizant technology solutions– ,developed multi cloud strategies in better using gcp (for its paas) and azure (for its saas)involved in loading and transforming large sets of the structured, semi-structured dataset and analysing them by running hive queriesdeveloped custom python program including ci/cd rules for google cloud data catalog for metadata managementdesign and develop spark job with scala to implement end to end data pipeline for batch processingdo fact dimensional modeling and proposed solution to load itprocessing data with scala, spark sql and load in hive partition tables in parquet file formatdevelop spark job with partitioned rdd (like hash, range, custom) for faster processingdevelop and deploy the outcome using spark and scala code in hadoop cluster running on gcpdevelop near real time data pipeline using flume, kafka and spark stream to ingest client data from their web log server and apply transformationperforms data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using erwin and mb mdrassist service developers in finding relevant content in the existing reference modelslike access, , oracle, flat files using connectors, tasks and transformations provided by aws data pipelineutilized spark sql api in pyspark to extract and load data and perform sql queriesworked on developing pyspark script to encrypting the raw data by using hashing algorithms concepts on client specified columnsresponsible for design, development, and testing of the database and developed stored procedures, views, and triggersdeveloped python-based api (restful web service) to track revenue and perform revenue analysiscompiling and validating data from all departments and presenting to director operationdevelop sqoop script and sqoop job to ingest data from client provided database in batch fashion on incremental basisuse distcp to load files from s3 to hdfs and processing, cleansing and filtering data using scala, spark sql, hive, impala query and load in hive tables for data scientists to apply their ml algorithms and generate recommendations as part of data lake processing layerbuild data pipelines in airflow in cp for etl related jobs using different airflow operators both old and newer operatorscreated big query authorized views for row level security or exposing the data to other teamsgood knowledge in using cloud shell for various tasks and deploying services. 06/2016to 12/2017big data engineerhoneywell– ,implemented a generic etl framework with high availability for bringing related data for hadoop & cassandra from various franchi sources using sparkexperience d in using platfora a data visualization tool specific for hadoop, and created various lens and viz boards for a real-time visualization from hive tablesqueried and analysed data from cassandra for quick searching, sorting and grouping through colimplemented various data modelling techniques for cassandrajoined various tables in cassandra using spark and scala and ran analytics on top of themparticipated in various upgrades and troubleshooting activities across enterprisesknowledge in performance troubleshooting and tuning hadoop clustersapplied spark advanced procedures like text analytics and processing using in-memory processingimplemented apache drill on hadoop to join data from sql and no sql databases and store it in hadoopcreated architecture stack blueprint for data access with nosql database cassandra;brought data from various sources into hadoop and cassandra using kafkaexperience d in using tidal enterprise scheduler and ooze operational services for coordinating the cluster and scheduling workflowsapplied spark streaming for real time data transformingcreated multiple dashboards in tableau for multiple business needsinstalled and configured hive and written hive udfs and used piggy bank, a repository of udf's for pig latinimplemented partitioning, dynamic partitions and buckets in hive for efficient data accessexported the analysed data to the relational databases using sqoop for visualization and to generate reports for the bi team using tableauimplemented composite server for the data virtualization needs and created multiple views for restricted data access using a rest apidevised and led the implementation of next generation architecture for more efficient data ingestion and processingcreated and implemented various shell scripts for automating the jobsimplemented apache sentry to restrict the access on the hive tables on a group levelemployed avro format for the entire data ingestion forecaster operation and less space utilizationexperience d in managing and reviewing hadoop log filesworked in an agile environment, and used rally tools to maintain the user stories and tasksworked with enterprise data support teams to install hadoop updates, patches, version upgrades as required and fixed problems, which were raised after the upgradesimplemented test scripts to support test-driven development and continuous integrationused spark for parallel data processing and better performancestools: map reduce, hdfs, hive, pig, impala, cassandra, spark, scala, solr, java, sql, tableau, pig, zookeeper, sqoop, teradata, centos, pentaho. 06/2014to 11/2015data engineeritc infotech–city.indiaworked on configuration and monitoring hadoop cluster using cloudera distributioninvolved in migrating data from on prem cloudera cluster to aws ec2 instances deployed on emr cluster and developed etl pipeline to extract logs and store in aws s3 data lake and further processed it using pysparkmoved files between hdfs and aws s3 and worked with s3 bucket in aws on a regular basisresponsible for developing data pipelines using flume, sqoop and pig to extract the data from weblogs and store them in hdfsmigrated data between various data sources like teradata, oracle, and mysql to hdfs by using sqoopused hcatalog to access hive table metadata from mapreduce and pig codedeveloped a data pipeline using kafka and storm for streaming data and to store it into hdfsused informatica powercenter for cleaning, managing, and integrating data from different sources for etl and loaded into a single warehouse repositoryused impala to read, write and query the hadoop data in hdfs from hbase and constructed impala scripts to reduce query response timeanalysed data stored in s3 buckets using sql, pyspark and stored the processes data in redshift and validated data sets by implementing spark componentsperformed etl operations using python, sql on many data sets to obtain metricsprepared data according to analyst requirements on the extracted data using pandas and numpy modules in pythoninvolved in designing and developing automation test scripts using pythoninvolved in writing multiple python scripts to extract data from different api’screated hbase tables using shell to load large sets of data from different databasesinvolved in scheduling time based oozie workflow engine to run multiple hive and pig jobsdeveloped flow xml files using apache nifi to process and ingest data into hdfsworked on performance tuning of apache nifi workflow to optimize the data ingestion speedsresponsible for collecting and aggregating large amounts of log data using flume and staging it into hdfs for further analysisworked on integration of apache storm with kafka to perform web analytics and upload streaming data from kafka to hbase and hiveresponsible for developing data pipelines using apache kafka by implementing kafka producers and consumersused hive optimization techniques like partitioning and bucketing to provide better performance with hiveql queriesloaded large amounts of data to hbase using mapreduce jobsworked on developing udfs to work with hive and wrote our tests in scalaused zookeeper to maintain configurations across clusters and for better synchronization, grouping and reliable distributed coordinationworked with kerberos and apache sentry for security and authorization on hadoopused git for version controltools: cloudera cdh4, hadoop 2.x (hdfs, mapreduce, yarn, sqoop, hive), aws, ec2, s3, redshift, impala, spark, pig, sql, hbase, kafka, zookeeper, flume, oozie, hcatalog, nifi, storm, informatica, python, mysql, scala, teradata, oracle, git. education expected in2013totobachelor’s:computer sciencecbit-hyderabad,gpa: skills technical skills hadoop/spark ecosystem:hadoop, mapreduce, hive/impala, yarn, kafka, flume, sqoop, oozie, zookeeper, sparkdatabases: sql server, postgresql, hbase, snowflake, cassandra, mongo dbcloud computing : amazon web services (aws), azure blob storage, azure data factory, azure synapse & google cloud platform(big query, big table, dataproc)bi tools: business objects xi, tableau 9.1, power biquery languages: /sql, t-sqlscripting languages: unix, pythonoperating systems: linux, windows, ubuntu, unixsdlc methodology: agile, scrum, waterfall, uml.