Listed below are some common roles and responsibilities that are expected to be performed by a PySpark Developer:
• Design, develop test, deploy, maintain and improve data integration pipeline
• Experience in Python and common python libraries
• Handling Data Warehousing/ Business Intelligence projects
• Knowledge of Hadoop technology
• Creating and loading tables in Hive tables
• Experience in optimizing SQL queries
• Innovate for data integration in Apache Spark-based Platform to ensure the technology solutions leverage cutting-edge integration capabilities