SummaryDynamic and motivated IT professional with around 15 years of experience as principal software developer having expertise in designing data intensive applications using Spark Ecosystem , Big Data Analytical , Cloud Data engineering , Data Warehouse Data Mart, Data Visualization , Reporting , and Data Quality solutions.Profound experience in performing Data Ingestion, Data Processing (Transformations, enrichment, and aggregations).Strong Knowledge on Architecture of Distributed systems and Parallel processing, In-depth understanding of Spark execution framework.Experienced with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context , Spark-SQL , Dataframe API and worked explicitly on Scala .Handled ingestion of data from different data sources into HDFS using Sqoop and perform transformations using Hive, Map Reduce and then loading data into HDFS.Experience of Partitions, bucketing concepts in Scala and worked on fine tuning the spark-scala and sql queries to optimize performance .Experience with different file formats like Avro , parquet , Json and XML .Expertise in Creating, Debugging, Scheduling and Monitoring jobs using Airflow.Experienced with using most common Operators in Airflow - Python Operator, Bash Operator.Hands-on experience in handling database issues and connections with SQL databases such as Netezza, Oracle, Redshift and PostgreSQL .Experience in designing and creating RDBMS Tables, Views, User Created Data Types, Stored Procedures.Expert in designing ETL data flows using creating mappings workflows to extract data from Netezza Oracle Servers.Expert in designing Parallel jobs using various stages like Join, Merge, Lookup, remove duplicates, Filter, Dataset, Lookup file set, Complex flat file, Modify, Aggregator, XML.Hands-on experience with Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto Scaling, CloudWatch, SNS, SES, SQS, Lambda, EMR and other services of the AWS family.Created and configured new batch job in Denodo scheduler with email notification capabilities and Implemented Cluster setting for multiple Denodo node and created load balance for improving performance activity.Instantiated, created, and maintained CI CD (continuous integration deployment) pipelines and apply automation to environments and applications.Worked on code versioning automation tools like GIT.Experienced with JSON based RESTful web services.Efficient Cloud Engineer with years of experience assembling cloud infrastructure. Utilizes strong managerial skills by negotiating with vendors and coordinating tasks with other IT team members. Implements best practices to create cloud functions, applications and databases. Skills Big Data Technologies Hadoop, MapReduce, HDFS, Sqoop, PIG, Hive, HBase, Oozie, Flume, NiFi, Kafka, Zookeeper, Yarn, Apache Spark, Mahout, Sparklib Databases Oracle, MySQL, SQL Server, MongoDB, Cassandra, DynamoDB, PostgreSQL, Teradata, Cosmos. Programming Python, PySpark, Scala, Java, C, C , Shell script, Perl script, SQL Cloud Technologies AWS, Microsoft Azure Frameworks Django REST framework, MVC, Hortonworks Tools PyCharm, Eclipse, Visual Studio, SQL Plus, SQL Developer, TOAD, SQL Navigator, Query Analyzer, SQL Server Management Studio, SQL Assistance, Eclipse, Postman Versioning tools SVN, Git, GitHub Operating Systems Windows 7 8 XP 2008 2012, Ubuntu Linux, MacOS Network Security Kerberos Database Modelling Dimension Modeling, ER Modeling, Star Schema Modeling, Snowflake Modeling Monitoring Tool Apache Airflow Visualization Reporting Tableau, ggplot2, matplotlib, SSRS and Power BI Machine Learning Techniques Linear Logistic Regression, Classification and Regression Trees, Random Forest, Associative rules, NLP and Clustering. ExperienceAWS Data Engineer,01 2022-02 2022Deloitte Rosslyn,CA,Designed and setup Enterprise Data Lake to provide support for various uses cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing dataResponsible for maintaining quality reference data in source by performing operations such as cleaning, transformation and ensuring Integrity in a relational environment by working closely with the stakeholders solution architect.Designed and developed Security Framework to provide fine grained access to objects in AWS S3 using AWS Lambda.Performed end- to-end Architecture implementation assessment of various AWS services like Amazon EMR, Redshift, S3Used AWS EMR to transform and move large amounts of data into and out of other AWS data stores and databases, such as Amazon Simple Storage Service (Amazon S3) and Amazon RDS.Used Spark SQL for Scala, Python interface that automatically converts RDD case classes to schema RDDImport the data from different sources like S3 HDFS into Spark RDD and perform computations using spark-scala spark-sql to generate the output responseCreating an automated alert notifications to identify and notify the idle EMR and EC2 clusters in our application regions to reduce the cost for EC2 and EMR resourcesDeveloped reusable framework to be leveraged for future migrations that automates ETL from RDBMS systems to the Data Lake utilizing Spark Data Sources and Hive data objectsDeveloped Grafana Dashboards based on the Log stash data and Integrated different source and target systems into Elasticsearch for near real time log analysis of monitoring End to End transactionsEnvironment AWS EMR, S3, RDS, Redshift, Lambda, Apache Spark, HIVE, SQOOP, Map Reduce, Python.Assessed organization technology infrastructure and managed cloud migration process.Implemented cloud policies, managed technology requests and maintained service availability.Data Engineer,01 2016-11 2019Splunk Memphis,TN,Worked on Azure Data Factory to integrate data of both on-prem (MY SQL, Cassandra) and cloud (Blob storage, Azure SQL DB) and applied transformations to load back to Azure SynapseManaged, Configured and scheduled resources across the cluster using Azure Kubernetes ServiceMonitored Spark cluster using Log Analytics and Ambari Web UITransitioned log storage from Cassandra to Azure SQL Datawarehouse and improved the query performanceInvolved in developing data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQLAlso Worked with Cosmos DB (SQL API and Mongo API)Develop dashboards and visualizations to help business users analyze data as well as providing data insight to upper management with a focus on Microsoft products like SQL Server Reporting Services (SSRS) and Power BIPerformed the migration of large data sets to Databricks (Spark), create and administer cluster, load data, configure data pipelines, loading data from ADLS Gen2 to Databricks using ADF pipelinesCreated various pipelines to load the data from Azure data lake into Staging SQLDB and followed by to Azure SQL DBCreated Databrick notebooks to streamline and curate the data for various business use cases and also mounted blob storage on DatabrickUtilized Azure Logic Apps to build workflows to schedule and automate batch jobs by integrating apps, ADF pipelines, and other services like HTTP requests, email triggers etcWorked extensively on Azure data factory including data transformations, Integration Runtimes, Azure Key Vaults, Triggers and migrating data factory pipelines to higher environments using ARM TemplatesIngested data in mini-batches and performs RDD transformations on those mini-batches of data by using Spark Streaming to perform streaming analytics in Data bricksEnvironment Azure SQL DW, Databrick, Azure Synapse, Cosmos DB, ADF, SSRS, Power BI, Azure Data lake, ARM, Azure HDInsight, Blob storage, Apache Spark.Adept in troubleshooting and identifying current issues and providing effective solutions.Managed performance monitoring and tuning while identifying and repairing issues within database realm.Identified key use cases and associated reference architectures for market segments and industry verticals.Designed surveys, opinion polls, and assessment tools to collect data.Tested, validated and reformulated models to foster accurate prediction of outcomes.Created graphs and charts detailing data analysis results.Recommended data analysis tools to address business issues.Developed new functions and applications to conduct analyses.Cleaned and manipulated raw data.Collaborated with solution architects to define database and analytics engagement strategies for operational territories and key accounts.Big Data Engineer Hadoop Developer,10 2013-12 2015General Dynamics Bossier City,MA,AnsibleDenodoDenodoCloudWatchAvroPySparkPySparkPySparkMLlibDataframeNiFiNiFiInteracted with business partners, Business Analysts and product owner to understand requirements and build scalable distributed data solutions using Hadoop ecosystemDeveloped Spark Streaming programs to process near real time data from Kafka, and process data with both stateless and state full transformationsWorked with HIVE data warehouse infrastructure-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HQL queriesBuilt and implemented automated procedures to split large files into smaller batches of data to facilitate FTP transfer which reduced 60 of execution timeWorked on developing ETL processes (Data Stage Open Studio) to load data from multiple data sources to HDFS using FLUME and SQOOP, and performed structural modifications using Map Reduce, HIVED eveloping Spark scripts, UDFS using both Spark DSL and Spark SQL query for data aggregation, querying, and writing data back into RDBMS through SqoopWritten multiple MapReduce Jobs using Java API, Pig and Hive for data extraction, transformation and aggregationAvrom multiple file formats including Parquet, Avro, XML, JSON, CSV, ORCFILE and other compressed file formats Codecs like gZip, Snappy, LzoStrong understanding of Partitioning, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performanceDeveloped PIG UDFs for manipulating the data according to Business Requirements and also worked on developing custom PIG LoadersDeveloping ETL pipelines in and out of data warehouse using combination of Python and Snowflakes SnowSQL Writing SQL queries against SnowflakeExperience in report writing using SQL Server Reporting Services (SSRS) and creating various types of reports like drill down, Parameterized, Cascading, Conditional, Table, Matrix, Chart and Sub ReportsUsed DataStax Spark connector which is used to store the data into Cassandra database or get the data from Cassandra databaseWrote oozie scripts and setting up workflow using Apache Oozie workflow engine for managing and scheduling Hadoop jobsWorked on implementation of a log producer in Scala that watches for application logs, transform incremental log and sends them to a Kafka and Zookeeper based log collection platformUsed Hive to analyze data ingested into HBase by using Hive-HBase integration and compute various metrics for reporting on the dashboardTransformed tPySpark using AWS Glue dynamic frames with PySpark cataloged the transformed the data using Crawlers and scheduled the job and crawler using workflow featureWorked on installing cluster, commissioning decommissioning of data node, name node recovery, capacity planning, and slots configurationDeveloped data pipeline programs with Spark Scala APIs, data aggregations with Hive, and formatting data (JSON) for visualization, and generatiPySparkEnvironment AWS, Cassandra, PySpark, Apache Spark, HBase, Apache Kafka, HIVE, SQOOP, FLUME, Apache oozie, Zookeeper, ETL, UDF, Map Reduce, Snowflake, Apache Pig, Python, Java, SSRSOnfidentialDeveloped and implemented Hadoop code while observing coding standards.Optimized and tuned Hadoop environments and modified hardware to meet prescribed performance thresholds.Developed new functions and applications to conduct analyses.Created graphs and charts detailing data analysis results.Tested, validated and reformulated models to foster accurate prediction of outcomes.Python Developer,09 2012-10 2013Fiserv City,STATE,AWS, S3, EC2, LAMBDA, EBS, IAM, Datadog, CloudTrail, CLI, Ansible, MySQL, Python, Git, Jenkins, DynamoDB, Cloud Watch, Docker, KubernetesLeveraged open communication, collective decision-making and thorough reviews to create performant and scalable systems.Worked with server-side and front-end technologies and leveraged common design patterns to code dynamic and user-friendly systems.Implemented new API routes, architected new ORM structures and refactored code to boost application performance.Harnessed version control tools to coordinate project development and individual code submissions.Introduced cloud-based technologies into Python development to expand on-premise deployment options.Wrote clear and clean code for use in projects.Resolved customer issues by establishing workarounds and solutions to debug and create defect fixes. Education and TrainingPost Graduate Data Engineering,Expected in02 2022-Purdue University-West Lafayette,INGPA Status-Post Graduate Data Science And Business Analytics,Expected in09 2021-University of Texas At Austin-Austin,TXGPA Status-Bachelor of Arts Business Administration And Management,Expected in12 2009-Califonia State University-Fullerton, CA,GPA Status-  SEP  Do you want to join one of the fastest-growing molecular diagnostics companies to help transform prenatal and oncology diagnostics that can affect millions of patient lives? BillionToOne, ranked at the top 5 of Y Combinator companies with 300M in funding from prominent VC firms, has developed a DNA molecular counter that increases cfDNA diagnostic resolution by over 1,000x. BillionToOne s first product, UNITY, is the first and only non-invasive prenatal test that can directly screen an unborn baby and assess its risk for common and severe recessive gene disorders and chromosomal abnormalities, as well as fetal sex and fetal antigens, from a single tube of blood from the pregnant mother without paternal testing. In 2023, BillionToOne launched two liquid biopsy products that will markedly improve oncology treatment and care. With Northstar Select, clinicians will be able to select cancer therapies that will be most effective to each patient based on their individual genetic makeup. Northstar Response will enable clinicians to monitor a patients early response to these cancer therapies, by measuring minute changes in their ctDNA, which can reduce ineffective treatments and unwanted toxicity, lower costs, and substantially improve clinical outcomes. We are currently looking for a Senior Software Engineer, Oncology to join our interdisciplinary oncology team. As a foundational member of the Oncology team, you will work closely with clinicians, scientists, computational biologists, and other engineers to help deliver world-class cancer diagnostics to oncologists and patients. If you thrive in a team setting, have an interest in biology and healthcare, have a knack for using technology to solve problems, and want to see your work push forward the cutting edge of cancer care, then this could be a dream job. This is a remote position, but you may also work from our office in Menlo Park or Union City, CA if youre locally-based. Projects you might work on include Developing data systems for classifying cancer genetics alternations and relevant treatment implications and clinical trials.Implementing an FDA-compliant laboratory information management system (LIMS) for robustly tracking specimens, reagents, and analyses throughout wetlab and drylab.Creating a convenient and easy to use web experience for physicians and patients receiving complex genomics information. Improving interoperability of EHR systems, especially with unstructured genetics information. Technologies we use Python, Django, Docker, Postgres, AWS, Terraform Requirements 5 years working with full-stack web technologies, data integration, andor data infrastructure.Values simplicity and pragmatism (you think radically but ship incrementally).Self-directed and comfortable working autonomously with peers across disciplines.Demonstrated interest in startups, biology, or healthcare. Nice-to-haves Experience working with computational biologists and other scientific or clinical disciplines strongly preferredDjango rest frameworkPrevious startup or entrepreneurial experience Benefits And Perks Working with a team of rockstars who bring out the best in everyoneOpen, transparent culture that includes weekly Town Hall meetingsThe ability to indirectly or directly change the lives of hundreds of thousand patientsMultiple medical benefit options employee premiums paid 100 of select plans, dependents covered at 80 Extremely generous Family Bonding Leave for new parents (16 weeks, paid at 100 )Retirement savings program including a 4 Company matchA variety of perks on campus (state of the art gym, food delivery, EV charging) At BillionToOne, we are proud to offer a combination of a (1) base pay range (actual amount offered is based on experience and salaryequity options split that the candidate chooses) (2) generous equity options offering, on top of (3) industry leading company benefits (free healthcare options, 401k match, very generous fully paid parental leave etc.). Please apply even if you dont meet every requirement, wed love to take a look at your profile! For this position, we offer a total compensation package of 265,000, including a base pay range of 180,000- 200,000 per year. BillionToOne is an equal opportunity employer. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.