21. distinct and dropduplicates in pyspark | how to remove duplicate in pyspark | pyspark tutorial
Channel: SS UNITECH
Date: 2023-06-14

Azure Databricks #spark #pyspark #azuredatabricks #azure 
In this video, I discussed how to use distinct & dropduplicate function in pyspark.

1. distinct function in pyspark
2. dropduplicate function in pyspark
3. dropduplicate vs distinct function in pyspark

Learn PySpark, an interface for Apache Spark in Python. PySpark is often used for large-scale data processing and machine learning.

1. pyspark introduction | pyspark tutorial for beginners | pyspark tutorial for data engineers:
https://youtu.be/hBDLfBILAuQ

2. what is dataframe in pyspark | dataframe in azure databricks | pyspark tutorial for data engineer:
https://youtu.be/VNNlNlVKn98

3. How to read write csv file in PySpark | Databricks Tutorial | pyspark tutorial for data engineer:
https://youtu.be/9kwxwCww4zI

4. Different types of write modes in Dataframe using PySpark | pyspark tutorial for data engineers:
https://youtu.be/-0_LkRtD3Bo

5. read data from parquet file in pyspark | write data to parquet file in pyspark:
https://youtu.be/B6wrbfLbaX0

6. datatypes in PySpark | pyspark data types | pyspark tutorial for beginners:
https://youtu.be/LqTUjOOHwQU

7. how to define the schema in pyspark | structtype & structfield in pyspark | Pyspark tutorial:
https://youtu.be/SqDlX_B7NmI

8. how to read CSV file using PySpark | How to read csv file with schema option in pyspark:
https://youtu.be/s1HHtTVg9xU

9. read json file in pyspark | read nested json file in pyspark | read multiline json file:
https://youtu.be/dOkPf_zVqaw

10. add, modify, rename and drop columns in dataframe | withcolumn and withcolumnrename in pyspark:
https://youtu.be/2SzrgwVhsy0

11. filter in pyspark | how to filter dataframe using like operator | like in pyspark:
https://youtu.be/4Hk8xmDPFZA

12. startswith in pyspark | endswith in pyspark | contains in pyspark | pyspark tutorial:
https://youtu.be/8Bep9kk4JB8

13. isin in pyspark and not isin in pyspark | in and not in in pyspark |  pyspark tutorial:
https://youtu.be/bY86Et-uIcA

14. select in PySpark | alias in pyspark | azure Databricks #spark #pyspark #azuredatabricks #azure
https://youtu.be/Ih9IlDO63CY

15. when in pyspark | otherwise in pyspark | alias in pyspark | case statement in pyspark:
https://youtu.be/d1GVRCXZ64o

16. Null handling in pySpark DataFrame | isNull function in pyspark | isNotNull function in pyspark:
https://youtu.be/si4bhjK1uB8

17. fill() & fillna() functions in PySpark | how to replace null values in pyspark | Azure Databrick:
https://youtu.be/OgAry0H_P9c

18. GroupBy function in PySpark | agg function in pyspark | aggregate function in pyspark:
https://youtu.be/_IaHywzYYFc

19. count function in pyspark | countDistinct function in pyspark | pyspark tutorial for beginners:
https://youtu.be/wDNSgMkkwPM

20. orderBy in pyspark | sort in pyspark | difference between orderby and sort in pyspark:
https://youtu.be/L3d6Eaxurz0

Azure Databricks Tutorial Platlist:
https://youtube.com/playlist?list=PLNRxk1s77zfgubs75vVMzHhPIhWqRo79C

Azure data factory tutorial playlist:
https://youtube.com/playlist?list=PLNRxk1s77zfjX_3ktp5sKsOh4Q2cWMMDX

ADF interview question & answer:
https://youtube.com/playlist?list=PLNRxk1s77zfgXfQKyScXtbn2MdFkvJtgH