Position: PySpark Data Engineer
Company: Invictus Gurus
Category: Data Engineer
Experience required: 5y
English level: upper
Published: 2022-09-01T00:00:00+03:00

We are looking for a PySpark Solutions Developer and Data Engineer capable of building and updating one of the most advanced, real-time global intelligence solutions in the banking and  financial sector worldwide.  This product works with the world’s largest banks and enterprises 
including Fortune 50. This is a high visibility, fast-paced key role that will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical banking and federal reserve systems.

Big Data & Data Engineer Key Responsibilities:
* Ability to design, build and unit test applications on Spark framework on Python.
* Build PySpark based applications for both batch and streaming requirements, which will  require in-depth knowledge on majority of Hadoop and NoSQL databases as well  (Elasticsearch, Cassandra, etc.).
* Develop and execute data pipeline testing processes and validate business rules and  policies.
* Optimize performance of the built Spark applications using configurations around Spark  Context, Spark-SQL, Data Frame, and Pair RDD's.
* Ability to design & build real-time applications using Spark Streaming.
* Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System,  HDFS File Types, HDFS compression codec.
* Build data tokenization libraries and integrate with Spark for column-level obfuscation.
* Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
* Create and maintain integration and regression testing framework on GIT repositories.
* Participate in the agile development process, and document and communicate issues and  bugs relative to data standards in scrum meetings.


Big Data & Data Engineer Requirements:
* Minimum 3 years of extensive experience in designing, building, and deployment of  PySpark-based applications.
* Expertise in handling complex large-scale Big Data environments preferably (20Tb+).
* Good implementation experience of OOPS concepts.
* Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.
* Ability to build abstracted, modularized reusable code components.
* Hands-on experience in generating/parsing XML, JSON documents, and REST API  request/responses.
* Able to quickly adapt and learn.
* Able to jump into an ambiguous situation and take the lead on resolution.
* Able to communicate and coordinate across various teams.
* Comfortable tackling new challenges and new ways of working.
* Ready to move from traditional methods and adapt into agile ones.
* Comfortable challenging your peers and leadership team.
* Can prove yourself quickly and decisively.
* Excellent communication skills and Good Customer Centricity.
* Strong Target & High Solution Orientation