this is my project report: Title: Implementing Naive Bayes Classifier for SMS Spam Detection using Hadoop MapReduce

Introduction:
The project aims to develop a robust SMS spam detection system using the Naive Bayes classification algorithm implemented on the Hadoop MapReduce framework. With the rise of unsolicited commercial messages (spam) overwhelming users' inboxes, an effective spam detection system is crucial for enhancing user experience and productivity.

Dataset:
The dataset used in this project consists of SMS messages labeled as either "spam" or "ham" (non-spam). The dataset is presented in a CSV file format, where the first column indicates the classification (spam or ham), and the second column contains the actual SMS message text.

Approach:
The project follows an iterative process to implement the Naive Bayes classifier using Hadoop MapReduce. The iterative process involves the following steps:

1. Data Preprocessing:
   - The dataset is loaded into the Hadoop Distributed File System (HDFS) for parallel processing.
   - Data cleaning and normalization techniques are applied to the SMS messages, such as removing special characters, converting to lowercase, and tokenizing the text into individual words.

2. MapReduce Implementation:
   - The project consists of three main components: NaiveBayesMapper.java, NaiveBayesReducer.java, and NaiveBayesDriver.java.
   - NaiveBayesMapper.java maps each input SMS message to (label, word) key-value pairs, where the label is either "spam" or "ham," and the word is a token from the message text.
   - NaiveBayesReducer.java calculates the word probabilities for both spam and ham classes by counting the occurrences of each word in each class. It also computes the prior probabilities of each class.
   - NaiveBayesDriver.java is the main driver program that sets up and runs the MapReduce job, specifying the input dataset, output directory, and the mapper and reducer classes.

3. Training and Classification:
   - The MapReduce job is executed on the Hadoop cluster, performing parallel processing of the dataset.
   - During the training phase, the Naive Bayes classifier learns the word probabilities and prior probabilities from the labeled dataset.
   - Once trained, the classifier can classify new SMS messages as spam or ham based on the learned probabilities using Bayes' theorem.

Output:
The output of the MapReduce job is a set of files containing the learned probabilities for each class (spam and ham). These files include:

1. Prior Probabilities: The output contains the prior probability of each class (spam and ham), which is the proportion of messages belonging to that class in the training dataset.

2. Conditional Probabilities: For each class, the output includes the conditional probabilities of each word given that class. These probabilities are calculated by dividing the count of a word in a particular class by the total number of words in that class, with smoothing techniques applied to handle unseen words.

The learned probabilities can be used to classify new SMS messages by computing the posterior probability of each class given the words in the message, and selecting the class with the highest probability as the predicted label.

Conclusion:
By leveraging the power of Hadoop MapReduce, this project implements the Naive Bayes classifier for SMS spam detection in a distributed and scalable manner. The iterative process involves data preprocessing, MapReduce implementation, training, and classification phases. The output files containing the learned probabilities can be used to classify new SMS messages effectively, contributing to an enhanced user experience by filtering out unwanted spam messages.