CROP RECOMMENDATION WITH MACHINE LEARNING: LEVERAGING ENVIRONMENTAL AND ECONOMIC FACTORS FOR OPTIMAL CROP SELECTION
Steven Sam and Silima Marshal D’Abreo
1, 2 Department of Computer Science College of Engineering, Design and Physical Science
Brunel University London [steven.sam@brunel.ac.uk](mailto:steven.sam@brunel.ac.uk)
Abstract
Agriculture constitutes a primary source of food production, economic growth and employment in India, but the sector is confronted with low farm productivity and yields aggravated by increased pressure on natural resources and adverse climate change variability. Efforts involving green revolution, land irrigations, improved seeds and organic farming have yielded suboptimal outcomes. The adoption of innovative computational solutions such as crop recommendation systems is considered as anew frontier to provide insights and help farmers adapt and address the challenge of low productivity. However, existing agricultural recommendation systems have predominantly focused on environmental factors and narrow geographical coverage in India, resulting in limited and robust predictions of suitable crops with both maximum yields and profits. This work incorporates both environmental and economic factors and 19 crop varieties across 15 states as input parameters to develop and evaluate two recommendation modules – Random Forest (RF) and Support Vector Machines (SVM) – using 10-fold Cross Validation, Time-series Split and Lag Variables approaches. Results show that the 10-fold cross validation approach produced exceptionally high accuracy, raising concerns of overfitting. However, the introduction of temporal order, which aligns more with real-world scenarios, reduces the model performance in the Time-series Split approach. To further increase the model accuracy while maintaining the temporal order, the Lag Variables approach was employed, which resulted in improved performance compared to the 10-fold cross validation approach. Overall, the models in the Time-series Split and Lag Variable Approaches offer practical insights by handling temporal dependencies and enhancing its adaptability to changing agricultural conditions over time. Consequently, the study shows the Random Forest model developed based on the Lag Variables as the most preferred algorithm for optimal crop recommendation in the Indian context. Key words : Crop recommendation model; Random forest; Support vector machines; Indian agriculture; Exploratory data analysis
1. Introduction
Agriculture is not only fundamental for food production but also constitutes a primary source for economic growth, employment and improvement of the wellbeing of many people globally. For
example, the World Bank reports that agriculture constitutes about 4% of the world's total gross domestic product (GDP), and in certain least developed nations, its contribution to GDP exceeds 25%. In India, agriculture accounts for 17% of total country’s GDP making it a critical sector for the country's
long-term and inclusive economic growth. The country leads in milk, spices, cotton, and Crop Recommendation with Machine Learning rice and wheat. India is also among the top exporters of rice, spices and meat. However, despite this significant development, farmers in India are confronted with low farm productivity and yields aggravated by increased pressure on natural resources (e.g., soil, air and
forest) and insufficient rainfall linked to adverse climate change variability. Approximately, only 45% of the areas occupied by 145 million small farm landholdings in India receive proper rainfall suitable for growing crops. The challenges of low productivity in Indian agriculture has put many people at the risk of experiencing food scarcity and financial difficulties, and farmers who are unable to repay loans taken from banks opt committing
suicides. Significant efforts are made by the government of India to address the challenge of low productivity including the green revolution, land reforms, irrigation facilities,
improved seeds, organic farming, water management and agriculture infrastructural investment. However, while these initiatives undeniably assist farmers throughout the agricultural cycle, the core challenge of low productivity persists particularly at the grassroots level. The adoption of innovative computational solutions such as crop recommendation systems is considered as anew frontier to help farmers adapt and address the challenge of low productivity. Crop recommendation system utilises algorithms and diverse data parameters such as weather data,
on farm data, soil nutrients and market information to forecast suitable crops with maximum yields and profits. Research shows that crop recommendation systems have been successfully applied to recommend crops to farmers
, detect plant disease, provide financial help, irrigation facilities and insurance to the farmers’ crops and maximise crop yield. Most crop recommendation systems incorporate one or more machine learning algorithms to make an efficient and accurate prediction of optimal crop to improve yield and profitability. For instance, Geetha et al., use Random Forest to generate a crop recommendation system with 97% accuracy. Kavita and Mathur implement a recommendation system to predict a crop with the highest yield using Decision Tree, Linear Regression, Lasso Regression and Ridge Regression. Decision Tree provides a higher accuracy of 98.62% compared to the other
algorithms. Lata and Chaudhari combine Random Forest, J48, Bayes Net and KStar to develop a recommendation system. Random Forest performs the best with an accuracy of 97.89%. In another study by Kumar et al., Support Vector Machine outperforms Random Forest, Decision Tree
Logistic Regression with a highest accuracy of 89.66% while Random Forest gave only 88%. Devi and
Selvakumari designs a recommendation system based on three algorithms: Naïve Bayes, Linear
Regression and Random Forest. Random Forest gave 99.09% accuracy compared with 90.9% of Naïve
Bayes. Collectively, research into crop recommendation systems is growing but challenges remain to make the recommendation models more robust and provide accurate forecasts to farmers in diverse
socio-economic and resource-constrained contexts. One of the most important challenges in developing accurate crop recommendation models involves selecting relevant input parameters or dataset for training the models. Without adequate, accurate and relevant data, even the most robust machine learning algorithms will not perform well to give the expected results. Across most of the studies, the environmental and economic features are the two broad categories of input parameters used separately for training crop recommendation models. For example, in a study conducted by Elomda S.
Sam & S.M. D’Abreo
et al., crops are suggested taking into consideration temperature, water and soil as the main features of the model. Jain and Ramesh developed a crop selection method to maximise crop yield and based on weather conditions and soil parameters like soil pH, soil nutrients and soil type while Patel and Patel developed their recommendation system considering only soil features like soil types, pH, electric conductivity, organic carbon, nitrogen, phosphorus, sulphur, zinc, boron,
iron, manganese and copper. Similarly, few researchers have also used the economic features, which constitute price, market conditions and crop cultivation costs, to develop models that recommend profitable crops to farmers. For instance,
Aadithya et al., developed a crop recommendation model including cost as an economic parameter to recommend the most suitable crop to the farmers. Filippi et al.’s,
recommendation model incorporated economic features such as the availability of tools and tractors,
their operating costs and the working speeds along with environmental features like the area available for cultivation, the sequence of operations required to grow each crop and the corresponding time windows to optimise agriculture profits. In the Indian context, most studies have almost exclusively focused on the environmental attributes with limited consideration for the economic parameters
. The objective of this paper is to develop a more comprehensive and inclusive crop recommendation system that incorporates the necessary input parameters for suggesting a wide range of optimal crops to farmers in India. There are approximately 145 million landholdings in India upon which diverse crops are cultivated. The small land size implies that farmers need appropriate knowledge and information to cultivate the right crops that generate high profitable yields in the face of the increasing climate change variability,
environmental challenges, market externalities and low productivity. To achieve this, we show that any recommendation system developed needs to be comprehensive and robust to forecast crop choices accurately for farmers across all geographical areas in India. We use eight environmental factors () along with major economic factors related to cost and price: operational cost, fixed cost, total cost of cultivation and minimum support price. In addition, the recommendation system involves 19 crops across 15 states as input parameters in India, extending the work of Pudumalar et al., and Reddy and Kumar
who have used a narrow range of crops (approximately five to ten crops) within a district or
state. 2. Method
This section describes the approaches for: (i) data collection and pre-processing, (ii) exploratory data analysis (iii) modelling, and (iv) performance evaluation. 2.1 Data Collection and pre-processing
We combined environmental and economic input parameters to develop and evaluate the accuracy of two machine-learning models (Random Forest and Support Vector Machines) for recommending high yield and profitable crops to farmers. The data were collected and merged from three sources: Kaggle for environmental parameters and the India Directorate of Economics and Statistics (DES) and Farmer’s Portal for economic parameters of cost and price. Following the work of Rani et al. 1
2 [https://eands.dacnet.nic.in/CostofCultivation.htm](https://eands.dacnet.nic.in/CostofCultivation.htm)
3 [https://farmer.gov.in/mspstatements.aspx](https://farmer.gov.in/mspstatements.aspx) Crop Recommendation with Machine Learning and other researchers, we consider abroad range of environmental parameters such as temperature, precipitation, wind speed and humidity along with soil attributes such as nitrogen, phosphorous, potassium (NPK) and soil type. These environmental features are important for improving crop production rates and determining soil
sustainability, and therefore provide necessary input parameters for crop recommendation systems
. Similarly, we included Operational Cost (e.g. labour, seed, fertilizer cost, etc.), Fixed
Cost (e.g. rental, capital, taxes, etc.) and Total Cost of crop cultivation along with the Minimum Support
Price (MSP) set by the Government of India for certain agricultural commodities annually as the economic input parameters. Following data collection, we developed a data dictionary for both the environmental parameters
and economic parameters to aid in pre-processing and analysis and maintain consistency and better understanding of the data. Table 1: Data dictionary – environmental parameters
Variable Name Variable Description Variable Type State Names State in which the crop is grown Character District Names District in which the crop is grown Character Crop Year Year in which the crop is grown Numeric Season Names Season in which the crop is grown Character Crop Names Name of the crop Character Area Area in hectares in which the crop is Numeric
grown
Temperature Temperature in the area where the crop is grown
Wind Speed Wind speed in the area where the crop is grown
Precipitation Precipitation in the area where the crop is grown
Humidity Humidity in the are where the crop is
grown
Soil Type Soil type in the area where the crop is
grown
N Nitrogen content in the soil where the crop is grown
P Phosphorous content in the soil where the crop is grown
K Potassium content in the soil where the crop is grown
Numeric
Numeric
Numeric
Numeric
Numeric
Numeric
Numeric
Numeric
Production Production of crop in tonnes Numeric
The pre-processing steps involved merging dataset, dealing with missing values and handling outliers and skewness to ensure that the data is in comprehensive and usable formats for training and testing the models.
2.1.1 Merging dataset
The dataset collected from the three sources were merged together in excel based on three common columns – State Names, Crop Year and Crop Names. Along with this, we had four more columns S. Sam & S.M. D’Abreo
(Operational Cost, Fixed Cost, Total Cost and MSP). As shown in Table 2, anew variable – ‘Yield’ was computed by dividing the variable Production by Area (Yield = Production / Area) to incorporate both these variables into a single variable and added to the final dataset. However, before combining the datasets, basic cleaning was done in excel to correct inconsistencies in the data like typos, inconsistent spellings and data entry errors to maintain uniformity.
It was essential that the spellings of the values in the variables - Crop Names and State Names are consistent across both the datasets as these were the common columns required to merge two datasets. For example, the state Chhattisgarh was spelled ‘Chhattisgarh’ in the data collected from Kaggle and ‘Chattisgarh’ in the data collected from
DES. In addition, the crop Bajra was incorrectly written as ‘Bajar’ in a few instances indicating a typo,
which was also corrected. Additionally, we noticed that the years 2011-15 were common crop years for both the datasets but the crop year 2015 had entries of only one State – Odisha, and was also removed from the dataset to avoid data bias. Table 2 : Data dictionary – economic parameters
Variable
Name
Variable Description Variable Type
Yield Yield of the crop cultivated Numeric Operational Operational cost of cultivation in Rs. per Numeric
Cost hectare
Operational cost of cultivation in Rs. per
hectare
Numeric
Fixed Cost Fixed cost of cultivation in Rs. per
hectare
Total Cost Sum of operational cost and fixed of cultivation in Rs. per hectare
MSP Minimum support price fixed by the
government
Numeric
Numeric
Numeric
The combined dataset from all the three sources was reduced to 12,839 instances with 20 columns due to the limitation of R Studio to handle large dataset and slow down analysis with numerous
features. It consists of 19 different crops across 15 states in India for the years 2011 14 (common years) which were incorporated in the crop recommendation model. The 19 crops are Arhar, Bajra, Barley, Cotton, Gram, Groundnut, Jowar, Jute, Maize, Moong, Paddy, Ragi, Rapeseed and
Mustard, Safflower, Sesamum, Soyabean, Sunflower, Urad and Wheat. The 15 states are - Andhra
Pradesh, Assam, Bihar, Chhattisgarh, Gujarat, Haryana, Karnataka, Madhya Pradesh, Maharashtra,
Odisha, Punjab, Tamil Nadu, Uttar Pradesh, Uttarakhand and West Bengal. 2.1.2 Handling Missing Values, Outliers and skewness
The feature ‘Production’ consisted of 89 missing instances randomly with no specific patterns. This made it difficult to compute any measure of central tendency. Additionally, it consisted of only a tiny portion of the total dataset and, therefore, were deleted from the dataset, as generally, if less than 5% of values are missing then it is acceptable to ignore them.
There were no duplicated instances in the dataset that needed cleaning. However, a boxplot analysis coupled with the summary statistics confirmed the presence of outliers in all the numerical variables except MSP. It was unclear if the outliers were implausible values or in fact correctly recorded instances that deviated from the other values in the dataset, as the outliers were a significant part of the dataset. Thus, considering removing outliers could lead to potential loss of important instances,
introduce bias and reduce the accuracy of the models, they were not discarded. Crop Recommendation with Machine Learning
Histograms and skewness results revealed some amount of skewness present in all the variables but the variable ‘Yield’ had extremely high positive skewness. This implies that the majority of instances have very low yields, while a few instances show exceptionally high yields. Given that the skewness of the variable ‘Yield’ highly deviated from the rest of the values and seemed erroneous, it was deleted from the dataset. This reduced the skewness as shown in Figure 2.
Figure 1: Skewness before removal Figure 2: Skewness after removal
In all, following data cleaning, the size of the dataset reduced to 12749 instances and 20 variables. 2.2 Exploratory Data Analysis
Exploratory Data Analysis (EDA) is an initial crucial step in the data analysis process that offers a systematic examination and visualisation of the data to gain insights, discover patterns, and uncover relationships among variables. In this study, we carried out EDA in RStudio to explore the connections among multiple variables through correlation analysis, assessment of
multicollinearity, and graphical examination. The objective was to identify key variables crucial for constructing the crop recommendation model and to eradicate less significant variables that could potentially lead to modelling issues. Detailed description and result of the EDA are presented in the result section below. 2.3 Modelling
We used R studio in this study to develop, train and validate Support Vector Machines and Random
Forest and algorithms for accurately recommending optimal crops to farmers. The models were trained and tested by splitting the dataset into 80% training and 20% testing data. This section below describes the algorithms and approaches used to develop the recommendation models, and the resulting accuracies are presented in the result section of the
paper. 2.3.1 Machine learning models
As observed in literature, Support Vector Machines and Random Forest offer complementary strength for crop recommendation systems. SVM can handle both linear and non-linear data through kernel
functions. They are particularly effective in high-dimensional spaces, making them suitable for complex datasets and mitigating the risk of overfitting due to their focus on maximising the margin between classes. In the context of crop recommendation, SVMs can be applied to capture complex interactions between factors such as climate, soil type, and crop attributes. However, computational complexity can become an issue, particularly with large
datasets. And SVM performance might degrade when dealing with noisy or overlapping classes, which is a common occurrence in real-world datasets. In contrast, RF models excel in their robustness and ability to handle large datasets without
overfitting. By aggregating multiple decision trees, they can provide high predictive accuracy and mitigate individual tree biases. It can handle missing values, and its insensitivity to irrelevant features further enhances its suitability for real-world applications. The model's feature importance metrics also aid in selecting relevant variables for crop recommendation systems. Their adaptability to both classification and regression tasks is valuable in addressing the multifaceted aspects of agricultural data, ultimately leading to improved recommendation accuracy. However, the ensemble nature of RF models can make them less
interpretable compared to individual decision trees and it might exhibit a bias towards categorical variables with more levels, affecting the accuracy of variable importance calculations. Further, the dataset's inherent characteristics, such as feature relationships and noise levels influence which model performs better. In all, considering the advantages and disadvantages of both the SVM and RF algorithms, we used them to build recommendation models, compared and evaluated the accuracy for forecast high yield and profitable crops for farmers. 2.3.2 Three Approaches used
The two algorithms –SVM and RF were trained using three approaches namely: 10-fold Cross
Validation Approach, Time-series Split Approach and Lag Variables
Approach. Approach 1 builds the models using 10-fold cross validation technique and randomly creates the train and split dataset as generated by the software. Both the training and testing dataset consisted of randomly shuffled selected instances from 2011 to 2014. In k-fold cross validation, the dataset was partitioned into k subsets and the model was trained and assessed k times,
with each fold serving as the validation set in turn.
The performance metrics from each fold were
averaged, offering an estimate of the model's overall effectiveness and reducing overfitting risk
. However, since the dataset used for the study has observations from the years 2011
to 2014 across five seasons – kharif, rabi, autumn, summer and winter – it was logical to treat the dataset as time-series data. Time series data is a collection of observations obtained through repeated measurements over time. Therefore, it was crucial to build models maintaining the temporal order of the data and capture the fluctuations that occur in the variables over time. In Time-series Split Approach 2, the observations were arranged in ascending order of year – 2011,
2012, 2013, 2014 and, within each year, instances were arranged based on seasons in the order –
winter, summer, kharif, autumn, rabi and whole year. Then, 80-20% train and test split were done in away that the developed models were trained using the data from past years and the test data mostly included instances from the year
2014. The split is a standard approach in time series analysis.
It aims to stimulate how well the model would perform in predicting future values based on patterns observed in historical data while avoiding the pitfalls of data leakage and ensuring that the temporal order is preserved. In the Lag variables Approach 3, along with the train and test split mentioned in Approach 2, lag features were created for Temperature, Precipitation, Humidity and Wind Speed based on the variable
Seasons and for Operational Cost, Total Cost and for MSP based on the variable Year. This is because it is revealed through the EDA process that environmental variables in the dataset tend to fluctuate Crop Recommendation with Machine Learning with the seasons while economic variables with the years. There is an evidently increasing pattern in cost and price variables over years, which is obvious considering the economic inflation every year. Creating lag variables is a fundamental technique in time series analysis that involves incorporating past values of a variable as features in a predictive model.
These lagged variables capture temporal dependencies and patterns, allowing the model to consider how the variable's history influences its future values and are crucial for addressing autocorrelation,
accounting for time delays in effects, and improving predictive accuracy. 2.4 Performance Evaluation
The developed models were evaluated with the help of accuracy value, Kappa statistics and F1 score. Accuracy measures the percentage of correct recommendations made by the model. Many studies have used accuracy as an evaluation metric. Figure 3: Accuracy Formula
However, the Kappa statistic (Cohen's Kappa) is most of the times preferred over accuracy because it accounts for chance agreement between predicted and actual outcomes. It is less sensitive to class imbalances and provides a more balanced measure of model performance, helping to avoid misleading assessments and hence will be used as one of the performance metrics. Kappa values range from -1 (no agreement) to 1 (perfect agreement). Additionally, accuracy often disregards the particular error types committed by the model and instead emphasizes overall correctness. It prioritizes a generalized correctness. To assess the model's competence in recognising and predicting True Positives, it is more appropriate to gauge F1 score that combines both precision and recall (EvidentlyAI, n.d). The F1 score is often referred to as the harmonic mean of precision and recall metrics and therefore is a single metric that weighs two ratios. Figure 4: F1 score Formula
Precision measures the proportion of correct positive recommendations (true positivies) out of all positive recommendations made by the model. S.
Sam & S.M. D’Abreo
Figure 5 : Precision Formula
Recall measures the proportion of correct positive recommendations out of all true positive instances in the data. Figure 6: Recall Formula
Hence, accuracy, kappa statistics and F1 scores were compared for the developed models through all the three approaches to recommend the most suitable modelling technique to the farmers to fulfil the research objective. Results
3.1 Exploratory Data Analysis Results
3.1.1 Scatter Plot Analysis
The scatter plot analysis discovered a significant positive relationship between five pairs of variables Temperature and Precipitation, Operational Cost and Total Cost, Total Cost and Fixed Cost, K and P
and Production and Area. The temperature and precipitation exhibit a negative relationship while all other pairs depict strong positive relationship. Figure 7a: Total Cost vs Fixed Cost Figure 7b: Precipitation vs Temperature Crop Recommendation with Machine Learning
Figure 7c: P vs K Figure 7d: Total Cost vs Operational Cost
Figure 7e: Production vs Area
The relationship between the variables can indicate the presence of multicollinearity. Multicollinearity
in machine learning algorithms can cause interpretability issues that might lead to the loss of reliability in determining the effects of individual features in the model. Hence, to confirm the presence of multicollinearity and mitigate the risk, we carried out correlation and multicollinearity
analysis. 3.1.2 Correlation and Multicollinearity Analysis
Results in Figures 8a and 8b show significant correlations between the five pairs of variables associated with crop recommendation prediction. The Temperature and Precipitation variables shared a strong negative correlation of -0.85 while the Total Cost and Operational Cost and Total Cost and Fixed Cost features illustrate strong positive correlation of 0.95 and 0.76 respectively. K and P also have a strong positive correlation of 0.87 followed by Production and Area with a positive correlation of 0.74. Arguably, the significant correlations among multiple variables suggest the presence of
multicollinearity between the variables, which can skew the results of the models. To confirm this, we carried out a multicollinearity test using the Variance Inflation Factor (VIF)
technique. S.
Sam & S.M. D’Abreo
Figure 8a : Correlation Plot Figure 8c : VIF values
Figure 8b : Correlation Values
The Variance Inflation Factor test evaluates multicollinearity in multiple linear regression models by measuring the extent to which the differing variance of the estimated regression coefficient is inflated
. VIF values < 5 suggests moderate multicollinearity while the VIF values ≥ 5 to 10 and above implies multicollinearity exists among the variables. This can be estimated as:
VIF =
1
1 – R
1 = Tolerance
In our study, a regression model was designed in RStudio using the numerical variables to test
multicollinearity amongst them. The result demonstrates that while some correlation exists, it is not severe enough to cause significant issues in terms of parameter estimation and model performance. The values of VIF all variables are 1< VIF ≤ 5 except for the Yield, Operational Cost Crop Recommendation with Machine Learning and Fixed Cost variables. The high values suggest that there is a problem of
collinearity with these three variables. However, based on the correlation values in Figures 8a and 8b,
the Yield variable does not have any strong correlations with other variables but still exhibits high VIF
value. This might be attributed to the fact that Yield is derived from two variables, namely Production and Area, which are strongly correlated. 3.1.3 Bar Plot Analysis
In the lag variables approach, it was essential to identify how different environmental and economic variables fluctuate with time-series variables – Years and Seasons. We used the bar plot analysis to capture these fluctuations. According to National Geographic, each season has its own
light, temperature and weather patterns, and these patterns repeat yearly. Hence, to understand how the environmental variables fluctuate with seasons, standard deviation barplots were plotted for each of the environmental variables - Temperature, Humidity, Wind Speed, and Precipitation against the variable – Season.
A careful observation of all these figures indicate that the bar heights of all these variables fluctuate with every season indicating seasonal fluctuations of environmental variables. Figure 9a : Temperature Fluctuations Figure 9b : Humidity Fluctuations with Seasons with Seasons
Figure 9c : Wind Speed Fluctuations Figure 9d : Precipitation with Seasons
Fluctuations with Seasons S. Sam & S.M. D’Abreo
Similarly, to understand the inflationary trends of economic variables yearly, mean barplots of economic variables – MSP, Operational Cost and Fixed Cost were plotted against each year and as seen, all the economic variables exhibit a yearly increasing pattern. Figure 9e : MSP increase Figure 8f : Operational Cost increase with Years with Years
Figure 9g : Fixed Cost increase with Years
In summary, the EDA results with the help of multicollinearity, correlation and scatter plot analysis identified the important variables for modelling and aimed to pinpoint highly correlated variables for tackling multicollinearity issues. Similarly, the barplot analysis established a relationship between multiple variables for successful formation of lag features. 3.2 Modelling Results
As mentioned in the methodology section, we used Support Vector Machines and Random Forest to develop the crop recommendation models using RStudio with three approaches – 10-fold Cross
Validation Approach, Time-series Split Approach and Lag Variables Approach. The results of the three approaches are described as follow:
3.2.1 10-fold Cross Validation Approach
The essence of the ten-fold cross-validation approach was to evaluate the performance of the models using the train and test split data. The dataset was first randomly shuffled before creating the train and test data. From the result, it is observed that RF performs better than SVM, although both exhibit high performance accuracy above 90%. Crop Recommendation with Machine Learning
Table 2: Performance metrics of 10-fold CV Approach
Models Accuracy Kappa
Random Forest 0.9966 0.9966
Support Vector Machine 0.9573 0.9538
Figure 8 displays the performance metrics of F1-score for each crop. The F1-score close to 1 indicates a strong performance with a good balance between precision and recall. Thus, as can be seen from
Figure 10, the RF model can make accurate recommendations for the several crop classes compare with SVM. RF displays approximately the performance metrics of 1 F1-score for all crops. Figure 10 : Approach 1 F1 scores for RF and SVM models
In sum, the results of the 10-fold cross-validation shows that the RF model exhibits remarkably high performance across the metrics including accuracy, kappa and F1-scores. However, given that accuracy and kappa statistics align closely in value raises a concern about potential overfitting. It is likely that the model may have captured noise in the training dataset and might not generalise well to new data or unseen data. To be confident about the given accuracy, it was essential to evaluate the performance of all these models on a completely new or unseen dataset as described in Approach 2. 3.2.2 Time-Series Split Approach
As outlined in the methodology, this approach arranged the observations in the chronological order of time and then partitioned the dataset into an 80% training and 20% testing ratio while maintaining the chronological sequence of instances, a common practice when handling time-series data to uphold the temporal coherence. Additionally, this partitioning ensured that the model is trained on historical observations while forecasting is done on completely unseen instances. S.
Sam & S.M. D’Abreo
Figure 11a: RF results Approach 2 Figure 11b: SVM results Approach 2
Figure 11c: Approach 2 F1 scores for RF and SVM models
Surprisingly, when the models are tested on completely unseen data in Approach 2, the accuracy,
kappa statistics and F1 scores drop significantly for both the models as shown in Figures 11a, band c. The drop in these values can be seen to provide a more realistic evaluation of the model's performance compared to the previous approach. In real-world scenarios, models need to be able to generalise to new, unseen data. If the model's performance drops because it struggles to adapt to temporal changes or new patterns in test data, it indicates that the model is not overly optimistic in its recommendation. This drawback necessitated for more accurate model adaptation and retraining strategies to better capture temporal patterns and shifts and is discussed in Approach 3. 3.2.3 Lag Variables Approach
Lag variables, also refer to time lag features, play a crucial role in capturing the historical patterns and temporal dependencies within the data through modelling relationships between past values and observations such as seasonality and long-term trends. In our study, this helps uncover recurring patterns and overarching trends that are critical for accurate crop recommendation. The barplot analysis in Figure12a confirms that environmental variables –
Temperature, Precipitation, Humidity and Wind Speed – fluctuate seasonally. Hence, we created lag variables for these variables based on seasons. Similarly, lag variables were also created based on Year Crop Recommendation with Machine Learning for economic variables – Operational Cost, Fixed Cost and for MSP – as the variables fluctuate with
years. Figure 12a: Creation of Lag features
As seen in Figure 12a, the number of lags for the variable Year was set to 5 while the number of lags for the variable Season was set to 7. We observed that any further creation of lags beyond these numbers failed to improve the accuracies of the model significantly. An evaluation of the results
Figures 12band c indicate an increase in the performance accuracy for the RF model from 78.55% to
83.62% while the SVM model increases from 71.18% to 74.38%. Comparatively, the kappa statistics also rise from 76.4% to 82.01% for the RF model and from 68.86% to 72.03% for the SVM model. Figure 12b: RF results Approach 3 Figure 12c: SVM results Approach 3
The F1 scores in Figure 12d also comparatively indicate better model performance overall in Approach
3 compared to approach 2. However, it is noteworthy that in this method too, RF surpasses SVM in terms of delivering superior model performance, achieving an accuracy rate of 83.62% compared to
SVM's 74.38%. S.
Sam & S.M. D’Abreo
Figure 13c: RF variable importance Approach 3
4.3 Performance evaluation metrics
Many researchers including Kapoor and Verma, Bondre and Mahagaonkar and Priyadarshini et al have traditionally relied solely on accuracy metrics to determine the most suitable crop recommendation model. Our study takes a more comprehensive approach by also incorporating Kappa statistics and F1 score for evaluating the developed models. The findings reported show that Kappa statistics is consistently lower than accuracy values for all the developed models across all the three approaches. For instance, in RF Approach 2, the accuracy stands at 78.55% whereas the Kappa score is 76.4%. This observation implies that while the accuracy may appear higher, the same model will give lower values of Cohen’s kappa when dealing with unbalanced datasets. To this end, it is quite likely that if researchers had incorporated Kappa statistics as well as a measure of performance evaluation, the accuracy of their developed models could have been reduced, as Kappa statistic provides a more balanced and reliable representation of model accuracy when adjusted for such class imbalances. Additionally, while researchers like Arooj et al. have applied precision (accuracy of positive crop recommendations) and recall (model’s ability to identify beneficial crops in relevant situations) independently as performance evaluation metrics. Our study goes beyond the use of F1 score and considers the trade-off between precision and recall, making it a useful single metric to evaluate the overall performance of the crop recommendation model. 5. Conclusion
This work proposed comprehensive crop recommendation models to help farmers in India optimise their crop selection decisions and reduce risk of crop failure while maximising profitability. Currently, most existing crop recommendation systems primarily rely on just environmental factors, cover limited geographical areas and recommend only a handful of crops thus creating a research gap. The model incorporates both environmental and economic parameters and 19 crop varieties across 15
different states. Two supervised machine-learning models (Random Forest and Support Vector Machines) were developed and evaluated using three approaches – 10-fold Cross Validation Approach, Time-series split approach and Lag variables approach. The models'
performance were evaluated using accuracy, Kappa statistics and F1 scores. Analysis reveals that the
10-fold cross validation approach produced exceptionally high accuracy,
raising concerns of overfitting. However, the introduction of temporal order, which aligns more with real-world scenarios, reduces the model performance in the Time-series Crop Recommendation with Machine Learning
Split approach. To further increase the model accuracy while maintaining the temporal order, the Lag
Variables approach was employed, which resulted in improved performance compared to the 10-fold cross validation approach. The models in the Time-series Split Approach and Lag Variable Approach offer practical insights by handling temporal dependencies and enhancing its adaptability to changing agricultural conditions over time. To this end, the study concludes that the Random Forest model developed based on the
Lag Variables is the most preferred algorithm for optimal crop recommendation in the Indian context. The model maintains the temporal order of the dataset and captures fluctuations through lag variables resulting in more realistic crop recommendations. However, future work should employ advanced approaches to explore complex pattern recognition and improve prediction accuracy. Additionally, combining multiple algorithms or hybrid models that integrate traditional time series methods with machine learning approaches can yield improved performance in crop recommendation. Furthermore, the study demonstrates the importance of considering both economic and environmental factors as input parameters for developing crop recommendation models. However, although these parameters are crucial and offer market considerations in the recommendation model, other important features need to be considered to develop robust recommendation models.
IEEE. Bhandari, B. P., Dhakal, S., & Tsou, C. Y.. Assessing the Prediction Accuracy of Frequency
Ratio, Weight of Evidence, Shannon Entropy, and Information Value Methods for Landslide Susceptibility in the Siwalik Hills of Nepal. Sustainability, 16, 2092. Bondre, D. A., & Mahagaonkar, S.. Prediction of crop yield and fertilizer recommendation using machine learning algorithms. International Journal of Engineering Applied Sciences and Technology, 4, 371-376. Devi, U., & Selvakumari, S.. Crop prediction and mapping using soil features with different machine learning techniques. In Proceedings of the International Conference on Innovative Computing & Communication (ICICC) . Dighe, D., Joshi, H., Katkar, A., Patil, S., & Kokate, S.. Survey of crop recommendation systems. IRJET, 5, 476-481. Doshi, Z., Nadkarni, S., Agrawal, R., & Shah, N.. AgroConsultant: intelligent crop S.
J. Intell. Technol. Appl. Stat, 11, 105-111. Ghosh, M.. Climate-smart agriculture, productivity and food security in India. Journal of
Development Policy and Practice, 4, 166-187. Gorain, S., Kuriachen, P., Kumar, C. V., & Suresh, A.. Land degradation and its impact on agricultural productivity: The case of India. Land Degradation & Development, 35, 196-212. Gulati, A., Paroda, R., Puri, S., Narain, D., & Ghanwat, A.. Food system in India. Challenges, performance and promise. Science and Innovations for Food Systems Transformation, 813-828.
Hayes, A. What is a time series and how is it used to analyze data? Available at:
](https://www.investopedia.com/terms/t/timeseries.asp) India Brand Equity Foundation (IBEF), Searching information about Indian agriculture..](https://www.ibef.org/industry/agriculture-india) Jain, S., & Ramesh, D.. Machine Learning convergence for weather based crop selection. In 2020 IEEE International Students' Conference on Electrical, Electronics and Computer Science (SCEECS). IEEE. Jaiswal, S., Kharade, T., Kotambe, N., & Shinde, S.. Collaborative recommendation system for agriculture sector. In ITM web of conferences. EDP Sciences. Kapoor, A., & Verma, A. K.. Crop selection using fuzzy logic-based expert system. Applications of Soft Computing for the Web, 121-137. Kanstrén, T.. A look at precision, recall, and f1-score: exploring the relations between machine
.] Kavita, M., & Mathur, P.. Crop yield estimation in India using machine learning. In 2020 IEEE 5th International Conference on Computing Communication and Automation (ICCCA).
IEEE. Kulkarni, N. H., Srinivasan, G. N., Sagar, B. M., & Cauvery, N. K.. Improving crop productivity through a crop recommendation system using ensembling technique. In 2018 3rd international conference on computational systems and information technology for sustainable solutions (CSITSS). IEEE. Kumar, Y. J. N., Spandana, V., Vaishnavi, V. S., Neha, K., & Devi, V. G. R. R.. Supervised machine learning approach for crop yield prediction in agriculture sector. In 2020 5th International Conference on Communication and Electronics Systems (ICCES). IEEE. Lata, K., & Chaudhari, B.. Crop yield prediction using data mining techniques and machine Crop Recommendation with Machine Learning learning models for decision support system. Journal of Emerging Technologies and Innovative Research (JETIR) . Pandian, S. K-fold cross validation technique and its essentials. Analytics Vidhya. Available at:
] Patel, K., & Patel, H.
B.. A state-of-the-art survey on recommendation system and prospective
extensions. Computers and Electronics in Agriculture, 178, 105779. Priyadharshini, A., Chakraborty, S., Kumar, A., & Pooniwala, O. R.. Intelligent crop recommendation system using machine learning. In 2021 5th international conference on computing methodologies and communication (ICCMC). IEEE. Pudumalar, S., Ramanujam, E., Rajashree, R. H., Kavya, C., Kiruthika, T., & Nisha, J.. Crop recommendation system for precision agriculture. In 2016 eighth international conference on advanced computing (ICoAC). IEEE. Rani, S., Mishra, A. K., Kataria, A., Mallik, S., & Qin, H.. Machine learning-based optimal crop selection system in smart agriculture.
Scientific Reports, 13, 15997. Reddy, D. J., & Kumar, M. R.. Crop yield prediction using machine learning algorithm. In 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS). IEEE. Sahoo, K., Samal, A. K., Pramanik, J., & Pani, S. K.. Exploratory data analysis using
Python. International Journal of Innovative Technology and Exploring Engineering (IJITEE), 8, 2019. Schafer, J. L., & Graham, J. W.. Missing data: our view of the state of the art. Psychological
methods, 7, 147. Shmueli, B.. Multi-Class Metrics Made Simple, Part III: the Kappa Score (aka Cohen’s Kappa
Coefficient). Medium [. Available at: https://towardsdatascience.com/multi-class-metrics-made-].] Shoaib, M., Shah, B., Ei-Sappagh, S., Ali, A., Alenezi, F., Hussain, T., & Ali, F.. An advanced deep learning models-based plant disease detection: A review of recent research. Frontiers in Plant Science, 14, 1158933. Shumway, R., & Stoffer, D.. Time series: a data analysis approach using R .
Chapman and
Hall/CRC. Talagala, T. S., Hyndman, R. J., & Athanasopoulos, G.. Meta-learning how to forecast time
series. Monash Econometrics and Business Statistics Working Papers, 6, 16. Tanasa, D., & Trousse, B.. Advanced data preprocessing for intersites web usage
mining. IEEE Intelligent Systems, 19, 59-65. Varghese, D.. Comparative study on classic machine learning algorithms. Medium. Available
[at : https://towardsdatascience.com/comparative-study-on-classic-machine-learning-]] Volpi, R., Namkoong, H., Sener, O., Duchi, J. C., Murino, V., & Savarese, S.. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31 . Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., ... & Yu, P.. Generalizing to unseen
domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering .