Job description:
novi labs delivers world-class oil & gas forecasts via an end-to-end platform that leverages machine learning and data science. join our talented technical team to help us improve and expand our products and further solidify our market-leading position. benefits of working at novi labs small, talented teamhighly collaborative work environment high autonomy with true project ownershipsmall company with visibility into company decisions and prioritieseasy line-of-sight to how your contributions directly benefit customers role we are looking for a seasoned data engineer that has a passion for sound design and engineering practices and is adept at making reasoned short-term vs long-term trade-offs. in this role you will be leading the buildout of our next generation data and machine learning platform, while helping to support and transition from our current systems. this is a key role on a small team. to be successful, it will be important that you feel comfortable driving projects on your own, but also enjoy brainstorming and collaboration. what youll do here after joining novi, you'll collaborate hand in hand with our team of data scientists and etl specialists. together, we're charting a course towards establishing an innovative greenfield platform that will unlock the power of machine-learning powered analytics products. as a key member of our team, you'll play a pivotal role in cultivating a culture of engineering excellence. your responsibilities will encompass the establishment of standards and the integration of emerging technologies to uphold the zenith of software engineering practices in python. your expertise will contribute to the improvement of our existing analytics platform, encompassing the implementation of novel functionalities and the enhancement of both the scalability and reliability of our data transformation and storage. this will be achieved through precise profiling and optimization techniques, coupled with the strategic design of cloud computing solutions. if you're driven by the prospect of making a profound impact within a collaborative and forward-thinking environment, we invite you to join us at novi. together, we'll continue to elevate the benchmark of data-driven solutions while nurturing your professional growth and development. required skills & experience minimum of 5 years of experience building robust and scalable data pipelinesstrong foundation in software engineering principles in pythonknowledge of the pydata stack (e.g. pandas) that goes deeper than the apisexperience handling, storing and linking multi-modal data typesexperience with the full software development lifecycle (git, ci, docker, test, etc.)strong database knowledge (sql & nosql)strong cloud architecture skills - aws preferred desired skills & experience interest in data science or previous experience working on a data science teamexperience working with gis dataprevious work on an agile teamexperience working for a start-up or small company cultural fit at novi, the culture fit is as important as the technical fit. our team will be a great fit for you if you: have a positive attitude and a growth mindsetenjoy working on small, highly collaborative teamsare self-motivated to improve your technical capacitiesinfluence without authority via well-reasoned ideas and clear communicationare comfortable working cross-functionally to deliver customer impactenjoy team and company successes as much as your owntake ownership of deliverables with strong attention to detailare happy to task switch when necessary to help teams succeedenjoy knowing whats happening across the entire company more about novi professional work environment with a clear understanding of worklife balanceflexible leave policyheathdentalvision insurance401k

Resume:
<<professional summaryover 8+ years of progressive professional experience in analysis, design, development and implementation as a sr. data engineer and data modeler/data analyst.strong experience in data migration, data cleansing, transformation, integration, data import, and data export.good knowledge in data quality & data governance practices & processes.highly qualified with about 5 years of experience in building big data applications, creating data lakes to manage structured and unstructured data.well versed with agile with scrum, waterfall model and test-driven development (tdd) methodologies.server reporting services (ssrs) and sql server integration services (ssis).worked with java based etl tool, talend and implemented integration solutions for cloud platforms with informatica cloud.extensive experience in setting up ci cd pipelines using tools such as jenkins, github nexus, and maven.experience developing on - premise and real time processes.experience on migrating sql database to azure data lake, azure data lake analytics, azure sql database, data bricks and azure sql data warehouse and controlling and granting database access and migrating on premise databases to azure data lake store using azure data factorystrong experience in big data technologies like spark sql, pyspark, hadoop, hdfs, hive, sqoop, flume, kafka, spark streamingexperience in job workflow scheduling and monitoring tools like oozie, m2.experience in importing and exporting data using sqoop from hdfs to relational database systems and vice-versa.experience in designing dashboards, reports, performing ad-hoc analysis and visualizations using tableau, power bi.hands-on experience on nosql databases like snowflake, hbase, cassandra and mongo db.experience in building and architecting multiple data pipelines, end to end etl and elt process for data ingestion and transformation in gcp and coordinating tasks among the team.expertise in dbms conceptsexperience in gcp dataproc, gcs, cloud functions, big table and big query.experience in writing and executing unit, system, integration and uatscripts in data warehouse projects.worked on different file formats like parquet, avro, orc and flat files.designing star schema, snowflake schema for data warehouse, ods architectureexperience with apache spark ecosystem using spark-core, sql, data frames and rdd's.experience d in data manipulation using python.hands-on experience working at amazon web services (aws) using elastic map reduce redshift, and ec2 for data processing, aws glue, rds, s3, lambda.proficient in installing, configuring and using apache hadoop ecosystems such as mapreduce, hive, pig, flume, yarn, hbase, sqoop, spark, storm, kafka, oozie, and zookeeper.experience in integrating kafka with spark streaming for high speed data processing.extensive experience in developing and driving strategic direction of sap operating system (sap ecc) and sap business intelligence (sap bi) systemdevelop effective working relationships with client teams to understand and support requirements, develop tactical and strategic plans to implement technology solutions, and effectively manage client expectations.an excellent team member with an ability to perform individually, good interpersonal relations, strong communication skills, hardworking and high level of motivation. skills technical skills hadoop/spark ecosystem:hadoop, mapreduce, hive/impala, yarn, kafka, flume, sqoop, oozie, zookeeper, sparkdatabases: sql server, postgresql, hbase, snowflake, cassandra, mongo dbcloud computing : amazon web services (aws), azure blob storage, azure data factory, azure synapse & google cloud platform(big query, big table, dataproc)bi tools: business objects xi, tableau 9.1, power biquery languages: /sql, t-sqlscripting languages: unix, pythonoperating systems: linux, windows, ubuntu, unixsdlc methodology: agile, scrum, waterfall, uml work history 10/2020tocurrentsr. data engineerhumana inc.– ,involved in requirements gathering, analysis, design, development, change management, deploymentinvolved in development of real time streaming applications using pyspark, apache flink, kafka, hive on distributed hadoop clusterutilized apache spark with python to develop and execute big data analytics and machine learning applications, executed machine learning use cases under spark ml and milibextracted data from heterogeneous sources and performed complex business logic on network data to normalize raw data which can be utilized by bi teams to detect anomaliesdesigned and developed flink pipelines to consume streaming data from kafka and applied business logic to massage and transform and serialize raw datadeveloped a common flink module for serializing and deserializing avro data by applying schemadeveloped spark streaming pipeline to batch real time data, detect anomalies by applying business logic and write anomalies to hbase tableimplemented layered architecture for hadoop to modularize designdeveloped framework scripts to enable quick developmentdesigned reusable shell scripts for hive, sqoop, flink and pig jobsstandardize error handling, logging and metadata management processesindexed processed data and created dashboards and alerts in splunk to be utilized/ action by support teamsresponsible for operations and support of big data analytics platform, splunk and tableau visualizationmanaged, developed, and designed dashboard control panel for customers and administrators using tableau, postgresql and rest api callsdesigned and developed applications using apache spark, scala, python, nifi, s3, aws emr on aws cloud to format, cleanse, validate, create schema and build data stores on s3developed ci- , and production environmentssupported production jobs and developed several automated processes to handle errors and notificationsalso, tuned performance of slow jobs by improving design and configuration changes of pyspark jobscreated standard report subscriptions and data driven report subscriptionstools: hadoop, map reduce, spark mllib, tableau, sql, excel, pig, hive, aws, postgresql, python, pyspark, flink, sql server 2012, t- -cd, git, xml, tableau. 01/2018to 09/2020sr. gcp data engineercognizant technology solutions– ,developed multi cloud strategies in better using gcp (for its paas) and azure (for its saas)involved in loading and transforming large sets of the structured, semi-structured dataset and analysing them by running hive queriesdeveloped custom python program including ci/cd rules for google cloud data catalog for metadata managementdesign and develop spark job with scala to implement end to end data pipeline for batch processingdo fact dimensional modeling and proposed solution to load itprocessing data with scala, spark sql and load in hive partition tables in parquet file formatdevelop spark job with partitioned rdd (like hash, range, custom) for faster processingdevelop and deploy the outcome using spark and scala code in hadoop cluster running on gcpdevelop near real time data pipeline using flume, kafka and spark stream to ingest client data from their web log server and apply transformationperforms data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using erwin and mb mdrassist service developers in finding relevant content in the existing reference modelslike access, , oracle, flat files using connectors, tasks and transformations provided by aws data pipelineutilized spark sql api in pyspark to extract and load data and perform sql queriesworked on developing pyspark script to encrypting the raw data by using hashing algorithms concepts on client specified columnsresponsible for design, development, and testing of the database and developed stored procedures, views, and triggersdeveloped python-based api (restful web service) to track revenue and perform revenue analysiscompiling and validating data from all departments and presenting to director operationdevelop sqoop script and sqoop job to ingest data from client provided database in batch fashion on incremental basisuse distcp to load files from s3 to hdfs and processing, cleansing and filtering data using scala, spark sql, hive, impala query and load in hive tables for data scientists to apply their ml algorithms and generate recommendations as part of data lake processing layerbuild data pipelines in airflow in cp for etl related jobs using different airflow operators both old and newer operatorscreated big query authorized views for row level security or exposing the data to other teamsgood knowledge in using cloud shell for various tasks and deploying services. 06/2016to 12/2017big data engineerhoneywell– ,implemented a generic etl framework with high availability for bringing related data for hadoop & cassandra from various franchi sources using sparkexperience d in using platfora a data visualization tool specific for hadoop, and created various lens and viz boards for a real-time visualization from hive tablesqueried and analysed data from cassandra for quick searching, sorting and grouping through colimplemented various data modelling techniques for cassandrajoined various tables in cassandra using spark and scala and ran analytics on top of themparticipated in various upgrades and troubleshooting activities across enterprisesknowledge in performance troubleshooting and tuning hadoop clustersapplied spark advanced procedures like text analytics and processing using in-memory processingimplemented apache drill on hadoop to join data from sql and no sql databases and store it in hadoopcreated architecture stack blueprint for data access with nosql database cassandra;brought data from various sources into hadoop and cassandra using kafkaexperience d in using tidal enterprise scheduler and ooze operational services for coordinating the cluster and scheduling workflowsapplied spark streaming for real time data transformingcreated multiple dashboards in tableau for multiple business needsinstalled and configured hive and written hive udfs and used piggy bank, a repository of udf's for pig latinimplemented partitioning, dynamic partitions and buckets in hive for efficient data accessexported the analysed data to the relational databases using sqoop for visualization and to generate reports for the bi team using tableauimplemented composite server for the data virtualization needs and created multiple views for restricted data access using a rest apidevised and led the implementation of next generation architecture for more efficient data ingestion and processingcreated and implemented various shell scripts for automating the jobsimplemented apache sentry to restrict the access on the hive tables on a group levelemployed avro format for the entire data ingestion forecaster operation and less space utilizationexperience d in managing and reviewing hadoop log filesworked in an agile environment, and used rally tools to maintain the user stories and tasksworked with enterprise data support teams to install hadoop updates, patches, version upgrades as required and fixed problems, which were raised after the upgradesimplemented test scripts to support test-driven development and continuous integrationused spark for parallel data processing and better performancestools: map reduce, hdfs, hive, pig, impala, cassandra, spark, scala, solr, java, sql, tableau, pig, zookeeper, sqoop, teradata, centos, pentaho. 06/2014to 11/2015data engineeritc infotech–city.indiaworked on configuration and monitoring hadoop cluster using cloudera distributioninvolved in migrating data from on prem cloudera cluster to aws ec2 instances deployed on emr cluster and developed etl pipeline to extract logs and store in aws s3 data lake and further processed it using pysparkmoved files between hdfs and aws s3 and worked with s3 bucket in aws on a regular basisresponsible for developing data pipelines using flume, sqoop and pig to extract the data from weblogs and store them in hdfsmigrated data between various data sources like teradata, oracle, and mysql to hdfs by using sqoopused hcatalog to access hive table metadata from mapreduce and pig codedeveloped a data pipeline using kafka and storm for streaming data and to store it into hdfsused informatica powercenter for cleaning, managing, and integrating data from different sources for etl and loaded into a single warehouse repositoryused impala to read, write and query the hadoop data in hdfs from hbase and constructed impala scripts to reduce query response timeanalysed data stored in s3 buckets using sql, pyspark and stored the processes data in redshift and validated data sets by implementing spark componentsperformed etl operations using python, sql on many data sets to obtain metricsprepared data according to analyst requirements on the extracted data using pandas and numpy modules in pythoninvolved in designing and developing automation test scripts using pythoninvolved in writing multiple python scripts to extract data from different api’screated hbase tables using shell to load large sets of data from different databasesinvolved in scheduling time based oozie workflow engine to run multiple hive and pig jobsdeveloped flow xml files using apache nifi to process and ingest data into hdfsworked on performance tuning of apache nifi workflow to optimize the data ingestion speedsresponsible for collecting and aggregating large amounts of log data using flume and staging it into hdfs for further analysisworked on integration of apache storm with kafka to perform web analytics and upload streaming data from kafka to hbase and hiveresponsible for developing data pipelines using apache kafka by implementing kafka producers and consumersused hive optimization techniques like partitioning and bucketing to provide better performance with hiveql queriesloaded large amounts of data to hbase using mapreduce jobsworked on developing udfs to work with hive and wrote our tests in scalaused zookeeper to maintain configurations across clusters and for better synchronization, grouping and reliable distributed coordinationworked with kerberos and apache sentry for security and authorization on hadoopused git for version controltools: cloudera cdh4, hadoop 2.x (hdfs, mapreduce, yarn, sqoop, hive), aws, ec2, s3, redshift, impala, spark, pig, sql, hbase, kafka, zookeeper, flume, oozie, hcatalog, nifi, storm, informatica, python, mysql, scala, teradata, oracle, git. education expected in2013totobachelor’s:computer sciencecbit-hyderabad,gpa: skills technical skills hadoop/spark ecosystem:hadoop, mapreduce, hive/impala, yarn, kafka, flume, sqoop, oozie, zookeeper, sparkdatabases: sql server, postgresql, hbase, snowflake, cassandra, mongo dbcloud computing : amazon web services (aws), azure blob storage, azure data factory, azure synapse & google cloud platform(big query, big table, dataproc)bi tools: business objects xi, tableau 9.1, power biquery languages: /sql, t-sqlscripting languages: unix, pythonoperating systems: linux, windows, ubuntu, unixsdlc methodology: agile, scrum, waterfall, uml.