The New York City Department of Education collects data on school bus delays and breakdowns to identify recurring issues and improve routes. You have data from the file bus.csv containing the following information:
School_Year: The year of the record.
Run_Type: Category of bus service where a delay or breakdown occurred.
Bus_No: Bus number.
Route_Number: Route number.
Reason: Reason for the delay or breakdown.
Occurred_On: Date of the incident.
Number_Of_Students_On_The_Bus: Number of students on the bus.
Exercise Objectives:
Data Loading and DataFrame Creation:
Q-Load the data from the text file and create a DataFrame.
Oky thirs i use colab as my notebook to work with this using pyspark and this basically 
The path of the data 
path="/content/bus.csv"




1- first we need to import the needed it packages 
2- instance a spark session as our entry point to work with spark 
2- next just read it the data as it’s and dislay a sample of first 5 lines of it oky 
Give the code to do it 
     # 1. Import necessary packages
from pyspark.sql import SparkSession
from pyspark.sql.types import *

# 2. Create a SparkSession
spark = SparkSession.builder.appName("BusDataAnalysis").getOrCreate()

# Define the schema (optional but recommended for better performance and data integrity)
schema = StructType([
    StructField("School_Year", StringType(), True),
    StructField("Run_Type", StringType(), True),
    StructField("Bus_No", StringType(), True),  # Bus_No can contain alphanumeric characters
    StructField("Route_Number", StringType(), True), # Route_Number may contain non-numeric characters
    StructField("Reason", StringType(), True),
    StructField("Occurred_On", StringType(), True),  # Initially load as String, will convert to Date later
    StructField("Number_Of_Students_On_The_Bus", IntegerType(), True),
])


# 3. Read the CSV data with the defined schema and specified options
path = "/content/bus.csv" 
df = spark.read.csv(path, header=True, schema=schema, inferSchema=False, multiLine=True, escape='"') #escape handles quotes within fields


# Display the first 5 rows
df.show(5,truncate=False) # truncate=False to show the full contents of longer strings


#  Further data cleaning and transformation (e.g., date parsing) can be done after this initial loading step.
# Example: Convert Occurred_On to DateType
# from pyspark.sql.functions import to_date
# df = df.withColumn("Occurred_On", to_date(df["Occurred_On"], "M/d/yyyy")) # Adapt date format if needed
   
content_copy Use code with caution.Python
Explanation and Improvements:
Schema Definition: Defining the schema upfront significantly improves performance, especially with larger datasets, because Spark doesn't need to infer the schema. I've made Bus_No and Route_Number StringTypes to accommodate potential non-numeric entries or inconsistencies.
inferSchema=False: This is crucial when you provide a schema. It tells Spark to use your provided schema, not try to infer one itself.
multiLine=True and escape: The multiLine=True option is added to handle potential multi-line fields within the CSV if they exist (e.g., Reason field having line breaks). The escape parameter is also added to correctly handle quotes within fields, which is a common cause of parsing errors. This assumes double quotes (") are used as escape characters. Adjust if a different escape character is used.
Date Handling: I've initially loaded Occurred_On as a StringType and then shown how to convert it to a DateType using to_date() in a commented-out section. This is best practice as it allows you to handle any date format inconsistencies more effectively after loading the data. Adapt the date format string (e.g., "M/d/yyyy" or "yyyy-MM-dd") according to the format in your CSV.
truncate=False: This is added to the show() function so you can see the full content of any potentially long string fields.
This improved code addresses potential data quality issues, provides more efficient processing, and prepares the DataFrame for further analysis and transformations. Remember to uncomment and adjust the date conversion code as needed for your specific date format.
Q- and this is the output oky ? 
+-----------+-----------------+------+------------+------------------+-------------------+------------------------------+
|School_Year|Run_Type         |Bus_No|Route_Number|Reason            |Occurred_on        |Number_Of_Strudents_On_The_Bus|
+-----------+-----------------+------+------------+------------------+-------------------+------------------------------+
|2015-2016  |Special Ed AM Run|48186 |N758        |Other             |2015-09-02T06:27:00|0                             |
|2015-2016  |Special Ed AM Run|2518  |L530        |Mechanical Problem|2015-09-02T06:24:00|0                             |
|2015-2016  |Special Ed AM Run|235   |K168        |Other             |2015-09-02T06:45:00|0                             |
|2015-2016  |Special Ed AM Run|2102  |K216        |Other             |2015-09-02T06:55:00|1                             |
|2015-2016  |Special Ed AM Run|48162 |N861        |Mechanical Problem|2015-09-02T06:55:00|0                             |
+-----------+-----------------+------+------------+------------------+-------------------+------------------------------+
only showing top 5 rows


Analysis of Delay/Breakdown Reasons:
Q-Identify the most common reasons for bus delays or breakdowns.
Oky so the question is asking us to find the most common reasons for the bus delays or breakdowns 
—> i think we need we need to count the number of delays or breakdowns by reason and sorting them by decreasing order 
→ then return the first row that represent the this information oky