Hyperspectral Vision-Based Machine Learning for Robust Plant Recognition in Autonomous Weed Control.

While herbicide application and mechanical cultivation remain the primary means for weed control in agricultural production, the only solution to-date for weed control within close proximity of crop plants in the seedline is hand hoeing. To reduce manual labor cost and minimize herbicide usage for organic farming, this research developed an intelligent robotic system for automated weed detection and control within the seedline. The system utilized visible and near infrared (NIR) reflectance-based features in hyperspectral images of plant foliage for real-time species recognition. This technique is less computationally intensive than the shape- and texture-based machine vision methods. For real-time, in-field applications, this technique is superior to traditional shape-based machine vision, as it does not require singulation of individual plants, and is robust to visual occlusion and less susceptible to leaf morphological variation or damage. A principle challenge of reflectance-based species recognition has been that the optical properties of plant foliage are functions of external growing conditions, and are greatly impacted by the variability in natural environment of agricultural fields. Prior studies of using visible and NIR spectroscopy for plant recognition were restricted to reflectance spectra measured on plants grown in a single season with exposure to a single environmental condition. This work, for the first time, demonstrated the potential of hyperspectral imaging technology for plant species identification under varying external factors of growing temperature, soil moisture and sunlight intensity, as well as over three multiple seasons of natural field environment. This work also developed adaptive learning techniques that mitigated environmental effects and provided solutions to robust plant recognition across the variation in the studied single conditions and seasons. Finally, the machine vision system was coupled with a thermal micro-dosing application system and validated under outdoor conditions for real-time automated weed control with heated food-grade oil. This research provided a complete solution to automated weed control in row crops using machine vision reflectance-based plant recognition. The technical results of this research are summarized as below: Multispectral Bayesian classifiers were developed for distinguishing tomatoes among black nightshade and pigweed. The effects of variation in the three single environmental factors of temperature, soil moisture, and solar irradiance, on spectroscopy-based plant recognition demonstrated that (i) the optimum performance, ranging from 88.2% to 95.3%, occurred when the models were applied in same environmental conditions as represented in training; (ii) increasing the deviation of the validation conditions from the calibration conditions degraded the performance to 62.5–81.7%; (iii) environmental stress made the plant species more distinguishable and slightly improved the overall accuracy by 1.3–6.9% for same condition applications; (iv) the Bayesian classifiers optimized for the normal conditions demonstrated more robust plant recognition over environmental variations. An environmentally-adaptive machine learning algorithm was developed for automatic site-specific recalibration of a Bayesian classifier in a dynamic environment. Validation performance of the classifier demonstrated that site-specific recalibration can be implemented by establishing the models exclusively with a fraction of new data (approximately 30 to 80 plants) without adverse impact due to ignoring the old data originally used to train the model. This method alleviated the bias produced by single-condition calibration and, overall, it improved the classification rates to 90.4–94.5% across variation in the three studied environmental factors. Global calibration was another approach investigated to improve robustness of the Bayesian classifiers to varying environmental conditions of the three studied factors. The overall classification rates of global classifiers ranged from 90.0% to 93.0% and the performance stability was also improved. Global calibration was recommended as it was able to provide robust classification performances across variation in the three studied environmental factors and was comparable to the optimal results obtained when the single-condition models were cross-validated on their training conditions. However, global calibration was not superior when used to discriminate tomatoes among weeds over three seasons (2005, 2006 and 2008), in which the plants were grown in a natural agricultural field environment. To improve the seasonal stability of plant recognition, a multiclassifier system was developed by integrating expert knowledge from historical data that most closely matched the new field environment. This method improved the performance of the global model by 10.5% (from 85.0% to over 95.5%) and provided an innovative direction for achieving robust plant recognition over the variation in field environment of multiple seasons. In an outdoor test of the complete weed control system, the hyperspectral vision system correctly mapped, by species, 91.0% of plant canopy and the thermal micro-dosing system then delivered preheated (160oC) food-grade oil exclusively to the targeted weed foliage. Fifteen days post application, the system successfully controlled 95.8% of black nightshade and 93.8% of pigweed; while only 2.4% of tomato was damaged to the extent of non-viability due to inadvertent spray.