SummaryDynamic and motivated IT professional with around 7 years of experience as a Big Data Engineer with expertise in designing data intensive applications using Hadoop Ecosystem , Big Data Analytical , Cloud Data engineering , Data Warehouse Data Mart, Data Visualization , Reporting , and Data Quality solutions .In - depth knowledge of Hadoop architecture and its components like YARN , HDFS, Name Node, Data Node, Job Tracker, Application Master, Resource Manager , Task Tracker and Map Reduce programming paradigm.Extensive experience in Hadoop led development of enterprise level solutions utilizing Hadoop components such as Apache Spark, MapReduce, HDFS, Sqoop, PIG, Hive, HBase, Oozie, Flume, NiFi, Kafka, Zookeeper, and YARN.Profound experience in performing Data Ingestion, Data Processing (Transformations, enrichment, and aggregations).Strong Knowledge on Architecture of Distributed systems and Parallel processing, In-depth understanding of MapReduce programming paradigm and Spark execution framework.Experienced with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context , Spark-SQL , Dataframe API , Spark Streaming, MLlib , Pair RDD s and worked explicitly on PySpark and Scala .Handled ingestion of data from different data sources into HDFS using Sqoop, Flume and perform transformations using Hive, Map Reduce and then loading data into HDFS.Managed Sqoop jobs with incremental load to populate HIVE external tables. Experience in importing streaming data into HDFS using Flume sources, and Flume sinks and transforming the data using Flume interceptors.Experience in Oozie and workflow scheduler to manage Hadoop jobs by Direct Acyclic Graph ( DAG ) of actions with control flows.Implemented the security requirements for Hadoop and integrating with Kerberos authentication infrastructure- KDC server setup, creating realm domain, managing.Experience of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance .Experience with different file formats like Avro , parquet , ORC , Json and XML .Expertise in Creating, Debugging, Scheduling and Monitoring jobs using Airflow and Oozie.Experienced with using most common Operators in Airflow - Python Operator, Bash Operator, Google Cloud Storage Download Operator, Google Cloud Storage Object Sensor.Hands-on experience in handling database issues and connections with SQL and NoSQL databases such as MongoDB , HBase , Cassandra , SQL server , and PostgreSQL .Created Java apps to handle data in MongoDB and HBase. Used Phoenix to create SQL layer on HBase.Experience in designing and creating RDBMS Tables, Views, User Created Data Types, Indexes, Stored Procedures, Cursors, Triggers and Transactions.Expert in designing ETL data flows using creating mappings workflows to extract data from SQL Server and Data Migration and Transformation from Oracle Access Excel Sheets using SQL Server SSIS .Expert in designing Parallel jobs using various stages like Join, Merge, Lookup, remove duplicates, Filter, Dataset, Lookup file set, Complex flat file, Modify, Aggregator, XML.Hands-on experience with Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto Scaling, CloudWatch, SNS, SES, SQS, Lambda, EMR and other services of the AWS family.Created and configured new batch job in Denodo scheduler with email notification capabilities and Implemented Cluster setting for multiple Denodo node and created load balance for improving performance activity.Instantiated, created, and maintained CI CD (continuous integration deployment) pipelines and apply automation to environments and applications.Worked on various automation tools like GIT, Terraform, Ansible. Experienced in fact dimensional modeling ( Star schema, Snowflake schema ), transactional modeling and SCD (Slowly changing dimension)Experienced with JSON based RESTful web services, and XML QML based SOAP web services and also worked on various applications using python integrated IDEs like Sublime Text and PyCharm .Efficient Cloud Engineer with years of experience assembling cloud infrastructure. Utilizes strong managerial skills by negotiating with vendors and coordinating tasks with other IT team members. Implements best practices to create cloud functions, applications and databases. Skills Big Data Technologies Hadoop, MapReduce, HDFS, Sqoop, PIG, Hive, HBase, Oozie, Flume, NiFi, Kafka, Zookeeper, Yarn, Apache Spark, Mahout, Sparklib Databases Oracle, MySQL, SQL Server, MongoDB, Cassandra, DynamoDB, PostgreSQL, Teradata, Cosmos. Programming Python, PySpark, Scala, Java, C, C , Shell script, Perl script, SQL Cloud Technologies AWS, Microsoft Azure Frameworks Django REST framework, MVC, Hortonworks Tools PyCharm, Eclipse, Visual Studio, SQL Plus, SQL Developer, TOAD, SQL Navigator, Query Analyzer, SQL Server Management Studio, SQL Assistance, Eclipse, Postman Versioning tools SVN, Git, GitHub Operating Systems Windows 7 8 XP 2008 2012, Ubuntu Linux, MacOS Network Security Kerberos Database Modelling Dimension Modeling, ER Modeling, Star Schema Modeling, Snowflake Modeling Monitoring Tool Apache Airflow Visualization Reporting Tableau, ggplot2, matplotlib, SSRS and Power BI Machine Learning Techniques Linear Logistic Regression, Classification and Regression Trees, Random Forest, Associative rules, NLP and Clustering. ExperienceAWS Data Engineer,01 2022-02 2022Deloitte Ann Arbor,MI,Designed and setup Enterprise Data Lake to provide support for various uses cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing dataResponsible for maintaining quality reference data in source by performing operations such as cleaning, transformation and ensuring Integrity in a relational environment by working closely with the stakeholders solution architectDesigned and developed Security Framework to provide fine grained access to objects in AWS S3 using AWS Lambda, DynamoDBSet up and worked on Kerberos authentication principals to establish secure network communication on cluster and testing of HDFS, Hive, Pig and MapReduce to access cluster for new usersPerformed end- to-end Architecture implementation assessment of various AWS services like Amazon EMR, Redshift, S3Implemented the machine learning algorithms using python to predict the quantity a user might want to order for a specific item so we can automatically suggest using kinesis firehose and S3 data lakeUsed AWS EMR to transform and move large amounts of data into and out of other AWS data stores and databases, such as Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDBUsed Spark SQL for Scala amp, Python interface that automatically converts RDD case classes to schema RDDImport the data from different sources like HDFS HBase into Spark RDD and perform computations using PySpark to generate the output responseCreating Lambda functions with Boto3 to deregister unused AMIs in all application regions to reduce the cost for EC2 resourcesImporting exporting database using SQL Server Integrations Services (SSIS) and Data Transformation Services (DTS Packages)Coded Teradata BTEQ scripts to load, transform data, fix defects like SCD 2 date chaining, cleaning up duplicatesDeveloped reusable framework to be leveraged for future migrations that automates ETL from RDBMS systems to the Data Lake utilizing Spark Data Sources and Hive data objectsConducted Data blending, Data preparation using Alteryx and SQL for Tableau consumption and publishing data sources to Tableau serverDeveloped Kibana Dashboards based on the Log stash data and Integrated different source and target systems into Elasticsearch for near real time log analysis of monitoring End to End transactionsImplemented AWS Step Functions to automate and orchestrate the Amazon SageMaker related tasks such as publishing data to S3, training ML model and deploying it for predictionIntegrated Apache Airflow with AWS to monitor multi-stage ML workflows with the tasks running on Amazon SageMakerEnvironment AWS EMR, S3, RDS, Redshift, Lambda, Boto3, DynamoDB, Amazon SageMaker, Apache Spark, HBase, Apache Kafka, HIVE, SQOOP, Map Reduce, Snowflake, Apache Pig, Python, SSRS, TableauAssessed organization technology infrastructure and managed cloud migration process.Configured computing, networking and security systems within cloud environment.Implemented cloud policies, managed technology requests and maintained service availability.Data Engineer,01 2016-11 2019Honeywell Huntsville,AL,Worked on Azure Data Factory to integrate data of both on-prem (MY SQL, Cassandra) and cloud (Blob storage, Azure SQL DB) and applied transformations to load back to Azure SynapseManaged, Configured and scheduled resources across the cluster using Azure Kubernetes ServiceMonitored Spark cluster using Log Analytics and Ambari Web UITransitioned log storage from Cassandra to Azure SQL Datawarehouse and improved the query performanceInvolved in developing data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQLAlso Worked with Cosmos DB (SQL API and Mongo API)Develop dashboards and visualizations to help business users analyze data as well as providing data insight to upper management with a focus on Microsoft products like SQL Server Reporting Services (SSRS) and Power BIPerformed the migration of large data sets to Databricks (Spark), create and administer cluster, load data, configure data pipelines, loading data from ADLS Gen2 to Databricks using ADF pipelinesCreated various pipelines to load the data from Azure data lake into Staging SQLDB and followed by to Azure SQL DBCreated Databrick notebooks to streamline and curate the data for various business use cases and also mounted blob storage on DatabrickUtilized Azure Logic Apps to build workflows to schedule and automate batch jobs by integrating apps, ADF pipelines, and other services like HTTP requests, email triggers etcWorked extensively on Azure data factory including data transformations, Integration Runtimes, Azure Key Vaults, Triggers and migrating data factory pipelines to higher environments using ARM TemplatesIngested data in mini-batches and performs RDD transformations on those mini-batches of data by using Spark Streaming to perform streaming analytics in Data bricksEnvironment Azure SQL DW, Databrick, Azure Synapse, Cosmos DB, ADF, SSRS, Power BI, Azure Data lake, ARM, Azure HDInsight, Blob storage, Apache Spark.Adept in troubleshooting and identifying current issues and providing effective solutions.Managed performance monitoring and tuning while identifying and repairing issues within database realm.Identified key use cases and associated reference architectures for market segments and industry verticals.Designed surveys, opinion polls, and assessment tools to collect data.Tested, validated and reformulated models to foster accurate prediction of outcomes.Created graphs and charts detailing data analysis results.Recommended data analysis tools to address business issues.Developed new functions and applications to conduct analyses.Cleaned and manipulated raw data.Collaborated with solution architects to define database and analytics engagement strategies for operational territories and key accounts.Big Data Engineer Hadoop Developer,10 2013-12 2015Infosys Ltd Irwindale,CA,AnsibleDenodoDenodoCloudWatchAvroPySparkPySparkPySparkMLlibDataframeNiFiNiFiInteracted with business partners, Business Analysts and product owner to understand requirements and build scalable distributed data solutions using Hadoop ecosystemDeveloped Spark Streaming programs to process near real time data from Kafka, and process data with both stateless and state full transformationsWorked with HIVE data warehouse infrastructure-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HQL queriesBuilt and implemented automated procedures to split large files into smaller batches of data to facilitate FTP transfer which reduced 60 of execution timeWorked on developing ETL processes (Data Stage Open Studio) to load data from multiple data sources to HDFS using FLUME and SQOOP, and performed structural modifications using Map Reduce, HIVED eveloping Spark scripts, UDFS using both Spark DSL and Spark SQL query for data aggregation, querying, and writing data back into RDBMS through SqoopWritten multiple MapReduce Jobs using Java API, Pig and Hive for data extraction, transformation and aggregationAvrom multiple file formats including Parquet, Avro, XML, JSON, CSV, ORCFILE and other compressed file formats Codecs like gZip, Snappy, LzoStrong understanding of Partitioning, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performanceDeveloped PIG UDFs for manipulating the data according to Business Requirements and also worked on developing custom PIG LoadersDeveloping ETL pipelines in and out of data warehouse using combination of Python and Snowflakes SnowSQL Writing SQL queries against SnowflakeExperience in report writing using SQL Server Reporting Services (SSRS) and creating various types of reports like drill down, Parameterized, Cascading, Conditional, Table, Matrix, Chart and Sub ReportsUsed DataStax Spark connector which is used to store the data into Cassandra database or get the data from Cassandra databaseWrote oozie scripts and setting up workflow using Apache Oozie workflow engine for managing and scheduling Hadoop jobsWorked on implementation of a log producer in Scala that watches for application logs, transform incremental log and sends them to a Kafka and Zookeeper based log collection platformUsed Hive to analyze data ingested into HBase by using Hive-HBase integration and compute various metrics for reporting on the dashboardTransformed tPySpark using AWS Glue dynamic frames with PySpark cataloged the transformed the data using Crawlers and scheduled the job and crawler using workflow featureWorked on installing cluster, commissioning decommissioning of data node, name node recovery, capacity planning, and slots configurationDeveloped data pipeline programs with Spark Scala APIs, data aggregations with Hive, and formatting data (JSON) for visualization, and generatiPySparkEnvironment AWS, Cassandra, PySpark, Apache Spark, HBase, Apache Kafka, HIVE, SQOOP, FLUME, Apache oozie, Zookeeper, ETL, UDF, Map Reduce, Snowflake, Apache Pig, Python, Java, SSRSOnfidentialDeveloped and implemented Hadoop code while observing coding standards.Optimized and tuned Hadoop environments and modified hardware to meet prescribed performance thresholds.Developed new functions and applications to conduct analyses.Created graphs and charts detailing data analysis results.Tested, validated and reformulated models to foster accurate prediction of outcomes.  SEP  Your Journey at Crowe Starts Here At Crowe, you have the opportunity to deliver innovative solutions to todays complex business issues. Crowes accounting, consulting, and technology personnel are widely recognized for their in-depth expertise and understanding of sophisticated process frameworks and enabling technologies, along with their commitment to delivering measurable results that help clients build business value. Our focus on emerging technology solutions along with our commitment to internal career growth and exceptional client value has resulted in a firm that is routinely recognized as a Best Place to Work. We are 75 years strong and still growing. Come grow with us! Job Description Crowe is seeking a talented and driven Senior Business Analyst in our Product Engineering practice. This individual will assume a business analyst role on a Scrum team that is responsible for assisting in the development and maintenance of the Ignite data analytics product that is leveraged by the Audit practice to deliver Crowes Audit services. Primary responsibilities include leading requirements elicitation with key stakeholders, documentation, story creation, and refinement for the team. This individual should have a passion for continuous improvement, innovation, and a history of successful software delivery and leadership within an Agile Scrum team. Responsibilities Ability to communicate effectively with all levels of the organization, from executive leadership to peers. Collaborate with Product Owner, stakeholders, and development teams to understand business goals and objectives. Ability to acquire a deep understanding of our products and systems, including their capabilities, features, and technical specifications. Perform analysis of large-scale epics for decomposition and work closely with Program Leadership and teams to ensure accurate scoping, feasibility assessment and effort estimation for epics and features. Conduct refinement sessions to decompose features into user stories, tasks, and acceptance criteria for the development team. Schedule, coordinate, and facilitate agile ceremonies and elicitation sessions with the Product Owner and stakeholders to identify, prioritize, and clarify requirements andor issues. Lead backlog management and refinement activities assisting in prioritizing and sequencing work items based upon business value and technical complexity. Collaborate within the Scrum team and Program Leadership on progress, providing updates and identifying potential risks or issues that surface during development. Develop and conduct product presentations andor demonstrations that effectively communicate the value, benefits, capabilities and features to various stakeholders and development team members. Partner with Development Lead to share Scrum Master responsibilities for the development team. Perform as a change agent on identified Agile process improvements for the development team. Perform change management activities in support of product releases. Collaborate with the QA Lead to ensure effective coordination and execution of User Acceptance Testing efforts. Collaborate across development teams to manage dependencies and ensure risks are proactively managed. Work with the development team to demonstrate work to the Product Manager and ensure it complies with the Definition of Done (DoD). Collaborate with cross-functional teams, including operations to promptly identify and prioritize issues reported in a production environment to ensure timely resolution and minimize impacts on users. Required Qualifications 5 years of BAExcellent analytical and critical thinking skillsStrong understanding of work requirement solicitation and decomposition techniquesExceptional learning agility and the ability to adapt quicklyAbility grasp business concepts and translate those to the development teamProficient level of understanding of Product Development and Software Development Life CyclesSolid experience with Agile Scrum methodologyExcellent communication skills, both verbal and written with the ability to effectively convey business and technical information to diverse audiences Demonstrated ability to work collaboratively and effectively with diverse stakeholders and team members as a part of a larger program fostering strong relationships and driving resultsProactive, self-starter, with a natural inclination toward taking individual initiative and personal accountabilityHigh degree of organization and the ability to manage multiple tasks effectively Preferred Qualifications Experience with Scaled Agile Framework (SAFe)Experience working with Azure DevOps or similar toolProduct Owner andor Scrum Master certificationExperience with SharePoint Microsoft 365 Power BIExperience with Data Driven or Data Heavy ProductsIndustry experience in professional services, finance, or accounting About the Team Product Engineering professionals at Crowe lead the software development efforts for the firm and build technology solutions in a consistent and reliable way. Product Engineering owns the software development competency for the firm, which is tied to standardized processes, practices, policies and controls for client-facing products and projects. Team members share expertise across various industries, targeting manufacturing and distribution, government, healthcare, financial services, and private equity groups based on demand from other Business Units. You will learn and build on a wide range of technologies in an environment where innovation and new ideas are highly encouraged. Hop on board to join one of the fastest growing technology consulting practices in the industry. Our Benefits At Crowe, we know that great people are what makes a great firm. We value our people and offer employees a comprehensive benefits package. Learn more about what working at Crowe can mean for you! How You Can Grow We will nurture your talent in an inclusive culture that values diversity. You will have the chance to meet on a consistent basis with your Career Coach that will guide you in your career goals and aspirations. Learn more about where talent can prosper! More about Crowe Crowe (www.crowe.com) is one of the largest public accounting, consulting and technology firms in the United States. Crowe uses its deep industry expertise to provide audit services to public and private entities while also helping clients reach their goals with tax, advisory, risk and performance services. Crowe is recognized by many organizations as one of the country s best places to work. Crowe serves clients worldwide as an independent member of Crowe Global, one of the largest global accounting networks in the world. The network consists of more than 200 independent accounting and advisory services firms in more than 130 countries around the world. Crowe LLP provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. Crowe LLP does not accept unsolicited candidates, referrals or resumes from any staffing agency, recruiting service, sourcing entity or any other third-party paid service at any time. Any referrals, resumes or candidates submitted to Crowe, or any employee or owner of Crowe without a pre-existing agreement signed by both parties covering the submission will be considered the property of Crowe, and free of charge.