Q- we choice this data Set to work with oky : 


About Dataset
Context
Welcome to the shopping world of Istanbul! Our dataset contains shopping information from 10 different shopping malls between 2021 and 2023. We have gathered data from various age groups and genders to provide a comprehensive view of shopping habits in Istanbul. The dataset includes essential information such as invoice numbers, customer IDs, age, gender, payment methods, product categories, quantity, price, order dates, and shopping mall locations. We hope that this dataset will serve as a valuable resource for researchers, data analysts, and machine learning enthusiasts who want to gain insights into shopping trends and patterns in Istanbul. Explore the dataset and discover the fascinating world of Istanbul shopping!
Content
Attribute Information:
invoice_no: Invoice number. Nominal. A combination of the letter 'I' and a 6-digit integer uniquely assigned to each operation.
customer_id: Customer number. Nominal. A combination of the letter 'C' and a 6-digit integer uniquely assigned to each operation.
gender: String variable of the customer's gender.
age: Positive Integer variable of the customers age.
category: String variable of the category of the purchased product.
quantity: The quantities of each product (item) per transaction. Numeric.
price: Unit price. Numeric. Product price per unit in Turkish Liras (TL).
payment_method: String variable of the payment method (cash, credit card or debit card) used for the transaction.
invoice_date: Invoice date. The day when a transaction was generated.
shopping_mall: String variable of the name of the shopping mall where the transaction was made.
Q- we do this 
"""
Q- this is a version Two which output the "Header" and "Infer The Schema" oky




"""
val df = spark.read.option("header","true").option("inferSchema","true").csv("dbfs:/FileStore/shared_uploads/amir21abdallah@gmail.com/customer_shopping_data.csv")
df.show(5)

As your Senior Data Scientist, here are seven data analysis questions derived from your Istanbul shopping malls dataset, categorized from easy to hard:
Easy Questions:
Sales Overview
What is the total revenue generated by each shopping mall from 2021 to 2023?
Customer Demographics
What is the distribution of customers across different age groups and genders?


Q- the first question is : 

—> we need to get the total revenue generated by each shopping mall from 2021 to 2023 


Q- before diving into the solution ? 

→ i need to get an view into how many type (categoties) of malls we have in this data oky ? 

→ the command will be like follow this logic 

Extract the column that contain the “malls “ 
Goup them by distinc value 
Sum them to get the number of dstincs onesoky 
Q- give the scala spark command here so 


Q- this the solution from “gemini”      import org.apache.spark.sql.functions._

// Load the DataFrame (assuming you've already done this)
val df = spark.read.option("header","true").option("inferSchema","true").csv("dbfs:/FileStore/shared_uploads/amir21abdallah@gmail.com/customer_shopping_data.csv")

// Calculate the number of distinct shopping malls
val distinctMallCount = df.select("shopping_mall").distinct().count()

// Print the result
println(s"Number of distinct shopping malls: $distinctMallCount")


// You can also display the distinct mall names if you want:
df.select("shopping_mall").distinct().show()
   
content_copy Use code with caution.Scala
Explanation:
df.select("shopping_mall"): This selects only the shopping_mall column from the DataFrame.
.distinct(): This removes duplicate values, so you get only the unique shopping mall names.
.count(): This counts the number of rows remaining after the distinct() operation, which gives you the number of distinct shopping malls.
println(...): This prints the result to the console.
The alternative using show() will display the actual mall names, which can be helpful for understanding your data. count() is more useful if you just need the numerical count.
This gives you the number of different shopping malls present in your dataset before you proceed to calculate the total revenue for each.

Q- but my solution look like this 

“
"""
Q- this "query" do not asnwer the right question but it's help us find later the global solution of the first problem oky
"""
val distinctMallCount = df.select("shopping_mall").distinct().count()


println(s"Number of distinct shopping malls: $distinctMallCount")


“ 
Number of distinct shopping malls: 10
command-3665228272098715:1: warning: a pure expression does nothing in statement position; multiline expressions may require enclosing parentheses
"""
^
distinctMallCount: Long = 10



Q- now we have know that we have 10 distinct “malls” in this dataset … now we need to group each mall by the number of revuniyse (sales * quantity ) in the whole data set periode 
Q- between 2021 and 2023 

Q- we can also sort them from the mall with the biggest number of sales to the lowest oky ? 


Q- how the logic you can format it ? 

→ we need to first group the “malls” and this feature “price*quantity” 
→ then apply an aggregation function like “sum” 
→ then sort them based on the alias “total_revunues” oky 


Q- as data engineering i want to be aware about the cost of queries i run so i need to check the cost of this query before i even implement 

→ Q- so is my order of execution is the most optimized order in to solve this query given the spark implementatio  
→ Q- second can use dummy data to manually see me the execution follow  i mean how goind from the original dataset into the finla solution oky