SummaryOver 5 years of engineering and finance experience as a Data Developer with cross platform integration experience using Hadoop and Spark architecture Hands-on experience configuring as well as installing Hadoop Ecosystem - HDFS, MapReduce, Pig, Hive, Oozie, Flume, HBase, Spark, Sqoop, Flume and Oozie. Strong understanding of various Hadoop services, MapReduce and YARN architecture. Vast knowledge in importing as well as exporting data in HDFS using SQOOP. Automated transfer of data from Hbase by developing Map Reduce jobs Expertise in analysis using PIG, HIVE and MapReduce. Experience in HDFS data storage and support for running map-reduce jobs. Involved in Infrastructure set up and installation of HDP stack on Amazon Cloud. Experienced with ingesting data from RDBMS such as SQL Teradata into HDFS using Sqoop and Oracle. Expertise in Hadoop architecture, HDFS, Map-reduce, Oozie, Sqoop, Spark, Hive, Zookeeper and NoSQL databases. Deployed and configured clusters in Cloudera Manager. Set up backups and disaster recovery for Data Node and Name Node metadata as well as sensitive data on clusters. Expertise in im plementing and designing HDFS access controls, directory and file permissions user authorization that facilitates stable, secure access for multiple users in a large multi-tenant cluster Knowledge on exporting as well as importing stream data into HDFS using Flume, Spark, and Kafka messaging systems. Utilized various schedulers on the Job tracker to share resources within clusters for Map Reduce jobs such as FIFO scheduler, Fair Scheduler and Capacity Scheduler. Monitored jobs with YARN, and provisioned, configured, and installed HBase, Kafka, Hive, Oozie, Sqoop, Ranger, Storm, Flume, Spark as well as maintained total architecture. AWS services for cloud migration such as S3, Redshift, EMR, Glue, Athena and DynamoDB. Expertise in Vertica DB architecture High Availability and column orientation. Great knowledge of Agile and Scrum methodologies, Behavior Driven Development, Domain Driven Design,Test Driven Development and continuous integration as well as delivery. Skilled in defining functional and gathering user interface requirements for applications and websites. Expertise in real time analysis by utilizing RDD, Datasets, Data Frames and Streaming API in Apache Spark. Expertise in using Spark Resilient distributed datasets and dataframe APIs over the Cloudera platform to perform analytics on Hive data by integrating Hadoop with Kafka. Expertise in uploading Click stream data from Kafka to HDFS. Expert in utilizing Kafka for messaging and publishing subscriber based messaging systems . Worked with NoSQL databases such as Cassandra, HBASE, PostgreSQL, MongoDB, Redis, DynamoDB SkillsSQL transactional replicationsSAN technologiesReverse engineering skillsWarehouse modelsSecurity ProtocolsEnterprise information architectureQuality analysisDeep learningData miningData analyticsCritical thinking ExperienceData Engineer,07 2021toCurrentPrincipal Financial Group Pittsburgh,PA, Analyze, design and build Modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes. Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. Created Pipelines in ADF using LinkedServices Datasets Pipeline to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards. Developed Spark applications using Pyspark andSpark-SQL for data extraction, transformation andaggregation from multiple file formats for analyzing transforming the data to uncover insights into thecustomer usage patterns. Responsible for estimating the cluster size, monitoringand troubleshooting of the Spark databricks cluster. Experienced in performance tuning of SparkApplications for setting right Batch Interval time,correct level of Parallelism and memory tuning.Adept in troubleshooting and identifying current issues and providing effective solutions.Managed performance monitoring and tuning while identifying and repairing issues within database realm.Contributed to maintaining Type databases in conjunction with data development and software engineering teams.Assisted solution providers with definition and implementation of technical and business strategies.Collaborated with solution architects to define database and analytics engagement strategies for operational territories and key accounts.Data Engineer,07 2020to06 2022Principal Financial Group Portland,OR,Worked on MongoDB by using CRUD (Create, Read, Update and Delete), Indexing, Replication, and Sharding featuresInvolved in designing the row key in HBase to store Text and JSON as key values in the HBase table and designed row key in such a way to get scan it in sorted orderOozie was integrated with the Hadoop stack which consists of Map-Reduce, Hive, Sqoop, and Pig and Unix shell scriptsWhile working on Hive tables used Hive QL, designed and Implemented Partitioning(Static, Dynamic) Buckets on HivePerformed Cache and Persist as well as Checkpointing when utilizing Spark Streaming APIs to build common learning data models to get near real-time data from Kafka and persisted into CassandraConducted cluster coordination services with Zookeeper and monitored the workload, capacity planning and job performance through Cloudera ManagerBuilt applications by utilizing Maven and integrated with Continuous Integration servers such as Jenkins to build jobsDeployed, maintained, and configured Test and multi-node Dev Kafka Clusters as well as handled clusters and implemented data ingestion for real time processing in KafkaCreated cubes in Talend for various aggregation types of data from PostgreSQL and MS SQL server to visualize dataMonitored Name Node health status in Hadoop as well as the number of Data Nodes and Task trackers running along with automating jobs to pull data from various MySQL data sources to push result set data to HDFSCreated story telling dashboards through Tableau Desktop to publish on Tableau Server and integrated GitHub for version control tools to maintain the versions in projectsDeployed Spark applications in python and utilized Datasets and DataFrames in Spark SQL for processing data fasterLoaded transactional data with Sqoop from Teradata, created managed and external tables in Hive, and worked with semi-structured and structured data of 5 Petabytes in sizeConstructed MapReduce jobs to validate, clean, and access data and worked with Sqoop jobs with incremental load to populate and load into Hive External tablesDesigned strategies to optimize distribution of weblog data over clusters, in addition to exporting and importing stored web log data into Hive and HDFS through SqoopResponsibilities of building scalable data solutions that are distributed through Hadoop and Cloudera as well as developed and designed automated test scripts in PythonIntegrated Apache Storm with Kafka to perform web analytics and to perform clickstream data from Kafka to HDFSDeveloped SQL scripts and designed solutions to implement Spark with Hive Generic UDFs for incorporating business logic within Hive queriesDeveloped data pipelines in AWS using S3, EMR, Redshift to extract data from weblogs to store into HDFSTransmitted streaming data from Kafka to HBase, Hive, and HDFS by integrating Apache Storm and wrote Pig scripts for transforming raw data from various data sources to form baseline dataParticipated in Agile meetings Ford Credit Customer Data domain, conducted daily scrum meetings and spring planningExpanded and optimized data pipelines and architecture as well as optimized data flow and collectionCreated pipelines from scratch using PySpark and scheduled jobs using AirflowStream processed data in Kafka streaming wrote Producer, Consumer, Connector and Streams API to handle stream of records, subscription of topics, consume input, and build reusable producers and consumersWorked with unstructured datasets such as IoT sources, sensor data, XML and JSON document sourcesWorked with on prem clusters as well as clusters on the cloud and used GCP Big Query, Data Fusion, and DataFlowScaled up architecture with Google Kubernetes and set up load balancingWorked on various optimization techniques in Spark such as cache and persist, using accumulators , bucketing and partitioning, garbage collection tuning, data serialization, windowing functions, and broadcast variablesUsed numerous Spark transformations such as groupByKey() reduceByKey(), flatMap(), filter(), sample(), union(), etcUtilized Dataproc to spin up clusters, Dataprep for data analysis, and Dataflow for streaming data using Apache BeamDealt with VPC controls on Google Cloud Platform and CMEK encryption for data securityEnvironment Hadoop, HDFS, HBase, Spark(Python and Scala), Azure, Databricks, Scala, Hive, Kafka, MapReduce, Sqoop, ETL, Java, Python, PostgreSQL, SQL Server, Teradata, Unix Linux.Big Data Developer,05 2017to05 2020Cognizant Technology Solutions Mount Laurel,NJ,Performed query tuning in HiveQL as well as performance tuning transformations in Pyspark using Spark RDDs and PythonUsed lambda functions to create a Serverless Data intake pipeline on AWSUsing python, constructed a Spark Streaming pipeline to receive real-time data from Apache Kafka and store it in DynamoDBImplemented Apache Spark data processing module to handle data from multiple RDBMS and Streaming sources, then compiled Apache Spark applications using Scala and PythonExtensive experience designing and scheduling multiple Spark Streaming batch Jobs in Python (pyspark) and ScalaAchieved high-throughput, scalable, fault-tolerant stream processing of live data streams using Apache Spark StreamingInvolved with the use for creating and saving data frames using various Python modules with pysparkSqooped data and performed Hive queries for data ingestion from relational databases to analyze historical dataExperienced with Elastic MapReduce (EMR) as well as setting up environments on amazon AWS EC2 instances for pipelines in AWSExpertise in handling Hive queries using Spark SQL such as window functions and aggregationsRan Spark applications on Docker using EMR and used AWS Glue data catalog as the metastore in Spark SQLConfigured different File Formats like Avro, parquet for HIVE querying and processing based on business logicUtilized Sequence files, RC files, Map side joins, bucketing, partitioning for Hive performance enhancement and storage improvementImplemented Hive UDF to implement business logic and performed extensive data validation using HiveInvolved in loading the structured and semi structured data into spark clusters using Spark SQL and Data Frames APIUtilized AWS CloudWatch to monitor the performance environment instances for operational and performance metrics during load testingScripting Hadoop package installation and configuration to support fully automated deploymentsInvolved in chef-infra maintenance including backup security fix on Chef ServerDeployed application updates using JenkinsInstalled, configured, and managed JenkinsTriggering the SIT environment build of the client remotely through JenkinsDeployed and configured Git repositories with branching, forks, tagging, and notificationsWorked on MongoDB database concepts such as locking, transactions, indexes, Shading, replication, schema designViewing the selected issues of web interface using SonarQubeDeveloped a fully functional login page for the company s user facing website with complete UI and validationsInstalled, Configured and utilized AppDynamics (Tremendous Performance Management Tool) in the whole JBoss Environment (Prod and Non-Prod)Installed and installed Hive in a Hadoop cluster and assisted business users application teams in fine-tuning their HIVE QL for optimal performance and efficient use of cluster resourcesUtilized Oozie workflow for ETL Process for critical data feeds across the platformConfigured Ethernet bonding for all Nodes to double the network bandwidthConfigured Kerberos Security Authentication protocol for existing clustersConstructed the use of Zookeeper failover controller (ZKFC) and Quorum Journal nodes for high availability for significant production clusters and automatic failover controller createdInstallation and deployment of many Apache Hadoop nodes on an AWS EC2 system, as well as development of Pig Latin scripts to replace the old traditional process with Hadoop, and data feeding to AWS S3Experience with AWS CloudFront, including the creation and management of distributions that provide access to an S3 bucket or an HTTP server running on EC2 instancesDeveloped Python scripts, UDFs using both Data frames SQL and RDD MapReduce in Spark 1.6 for Data Aggregation, queries and writing data back into OLTP system through Sqoop And Developed enterprise application using PythonConstructed Spark application performance optimization, including determining the appropriate Batch Interval time, Parallelism Level, and Memory TuningExperience and hands-on knowledge in Akka and LIFT FrameworkUsed PostgreSQL and No-SQL database and integrated with Hadoop to develop datasets on HDFSEnvironment HDFS, Map Reduce, Hive 1.1.0, Kafka, Hue 3.9.0, Pig, Flume, Oozie, Sqoop, Apache Hadoop 2.6, Spark, SOLR, Storm, Cloudera Manager, Red Hat, MySQL, Prometheus, Docker, Puppet, YARN,, Spark-SQL, Python, Amazon AWS, Elastic Search, Tableau, LinuxKey Skills SQL, Apache hive, Apache Spark(Databricks, Jupyter Notebook), Anaconda, Python (Django, Pandas, Flask, Keras, NumPy, Scikit-learn, MatPlotLib, Tensorflow, ), Time Series Forecasting, A B testing, Bayesian methods, PowerBI, Microsoft(Word, Excel, Powerpoint), Java, Data Visualization, Analytical Skills, Cost Accounting, Corporate Finance Knowledge, Statistical Analysis(RStudio) Education and TrainingBachelor s Computer Science,Expected intoUttara Institute of Business and Technology-,GPA  SEP  When you join Verizon Verizon is one of the world s leading providers of technology and communications services, transforming the way we connect around the world. We re a human network that reaches across the globe and works behind the scenes. We anticipate, lead, and believe that listening is where learning begins. In crisis and in celebration, we come together-lifting up our communities and striving to make an impact to move the world forward. If you re fueled by purpose, and powered by persistence, explore a career with us. Here, you ll discover the rigor it takes to make a difference and the fulfillment that comes with living the NetworkLife. What you ll be doing... As a Sr Engr Cslt - Data Engineer within the Artificial Intelligence and Data Organization (AI D), you will manage various activities including artificial intelligence, data engineering, operations automation, security remediation and industry leading Omni channel technology to improve the efficiency, customer experience and profitability of the company. You will analyze marketing, customer experience and digital operations environments to build data pipelines, transform data into actionable intelligence. You will turn raw data into usable data pipelines and build data tools and products for effort automation and easy data accessibility. At Verizon, we are on a multi-year journey to industrialize our data science and AI capabilities. Very simply, this means that AI will fuel all decisions and business processes across the company. At 130B dollars in annual revenue, this is a pioneering opportunity to design and shape AI at scale in the Telco industry. With our leadership in bringing 5G network nationwide, the opportunity for AI will only grow exponentially in going from enabling billions of predictions to possibly trillions of predictions that are automated and real-time Manage all marketing related projects on Big DataCloud platform including design of new applications and training other developers. Gather requirements, assess gaps, and build roadmaps and architectures to help the analytics driven organization achieve its goals. Work closely with Data Analysts to ensure data quality and availability for analytical modeling. Identify gaps and implement solutions for data security, quality, and automation of processes. Support maintenance, bug fixes and performance analysis along the data pipeline. Design, build, document, test and implement new data pipelines and analytics. The job involves designing and building scalable data pipelines to provide key insights to the front-end applications and agents for quick decision making and effective channel orchestration. Collaborate in cross-functional teams to source new data, develop schema requirements, and maintain metadata Build semantic layer (curated facts), dimension tables for reports and analytics Coordination with the offshore team for design and implementation. Identify ways to improve data reliability, efficiency and quality Use large data sets to address business issues Use data to discover tasks that can be automated Design and implement fact tables, reports and dashboards using Tableau, Looker and other tools Analyze existing Teradata SQL queries for performance improvement. What we re looking for... You re curious about new technologies and the game-changing possibilities it creates. You like to stay up-to-date with the latest trends and apply your technical expertise to solve business problems. You thrive in a fast-paced, innovative environment working as a phenomenal teammate to drive the best results and business outcomes. You ll need to have Bachelor s degree or four or more years of work experience. Four or more years of relevant work experience. Even better if you have one or more of the following 3 years of Database experience in Teradata SQL, Teradata Utilities, SQL Server, SSIS, and OLAP 3 years of experience in designing, building, and deploying production-level data pipelines using tools from Hadoop stack (HDFS, Hive, Spark, Streaming, HBase, Kafka, Oozie, NiFi etc.) and programming in PythonScala 1 year(s) of experience in Cloud data engineering, preferably Google Cloud with Big Query Dashboard development experience in Tableau, Qlik andor Looker A degree. Master s degree in Computer Science, Information Systems and or related technical discipline. Teradata Big Data Analytics Certification Knowledge of telecom industry Experience to working with a distributed team Ability to effectively communicate through presentation, interpersonal, verbal and written skills. Why Verizon? Verizon is committed to maintaining a Total Rewards package which is competitive, valued by our employees, and differentiates us as an Employer of Choice. We are a pay for performance company and your contribution is rewarded through competitive salaries, performance-based incentives and an employee Stock Program. We create an opportunity for us all to share in the success of Verizon and the value we help to create through this broad-based discretionary equity award program. Your benefits are market competitive and delivered by some of the best providers. You are provided with a full spectrum of health and wellbeing resources, including a first in-class Employee Assistance Program, to empower you to make positive health decisions. We offer generous paid time off benefits to help you manage your work life balance and opportunities for flexible working arrangements . Verizon provides training and development for all levels, to help you enhance your skills and develop your career, from funding towards education assistance, award-winning training, online development tools and access to industry research. You will be able to take part in volunteering opportunities as part of our environmental, community and sustainability commitment. Your benefits package will vary depending on the country in which you work. subject to business approval If Verizon and this role sound like a fit for you, we encourage you to apply even if you don t meet every even better qualification listed above. This role is eligible to be considered for the Department of Defense SkillBridge Program. Where you ll be working In this hybrid role, you ll have a defined work location that includes work from home and assigned office days set by your manager. Scheduled Weekly Hours 40 Equal Employment Opportunity We re proud to be an equal opportunity employer - and celebrate our employees differences, including race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, and Veteran status. At Verizon, we know that diversity makes us stronger. We are committed to a collaborative, inclusive environment that encourages authenticity and fosters a sense of belonging. We strive for everyone to feel valued, connected, and empowered to reach their potential and contribute their best. Check out our diversity and inclusion page to learn more.