# Data management in precision farming

Pierre Blavy

► To cite this version:

Pierre Blavy. Data management in precision farming. Data management in precision farming (Data management in precision farming), 2017. hal-02788369

HAL Id: hal-02788369

https://hal.inrae.fr/hal-02788369

Submitted on 5 Jun 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Data management in precision farming

Pierre Blavy, INRA

12 december 2017

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

* What can I do with existing data? (meta analysis)

* What's the question ?

⇒ Will my data answer it (write the method+stats first)

<figure><img src="image_4.png" /></figure>
## Practical decision

* Real time (Important for large farms)

* On farm alerts (vêlage, chaleurs, ...)

* Long term

* Herd management tools

* Planning assistants

## V.S. Research question

* Impact of feeding on production/environment

* Gentic selection on food efficiency?

* Animal behaviour/welfare?

<figure><img src="image_5.png" /></figure>
# Costs v.s. benefits

<figure><img src="image_6.png" /><figcaption>Plonk et Replonk</figcaption></figure>

## Cost v.s. benefits tradeoff

* money

* time

* human factors

<figure><img src="image_7.png" /></figure>
MODEL CALCULATIONS

"Garbage In-garbage Out" Paradigm

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

* Farm proof systems (machine robustness)

* Know what you measure

* What's the data (definition, format, unit)

* In which context it's measured / usable ?

<figure><img src="image_10.png" /></figure>
* Use databases (SQL is your friend)

* SQL https://www.w3schools.com/sql/

* SQLite in R http://cpc.cx/iZj

* Maximize automation (program boring repetitive stuff)

* With R http://cpc.cx/kQJ

* With python http://cpc.cx/kQK

* Others softwares are OK

* Write documentation

* Someday, it will save someone

* Who may be you.

<figure><img src="image_11.png" /></figure>
Watch your data

* Make plots

► R http://cpc.cx/kQL

► R+ggplot2 http://cpc.cx/kQM

► R+shiny https://shiny.rstudio.com/

* Descriptive stats

► mean, median, in, max...

► proportion of missing values

► R summary

* PCA...

<figure><img src="image_12.png" /></figure>
# Principal Component Analysis

* PCA maximize the projected variance

* Orhtogonal axis

* R, factominer http://cpc.cx/kQQ

* PCA is a good tool for

* Summarizing data (check inertia)

* Finding outliers (check individual weight)

* Constructing synthetic variables (check axis contribution)

* PCA is NOT a tool for

* Acurate correlations ⇒ watch the correlation matrix

* Testing hypothesis ⇒ statistical test

<figure><img src="image_13.png" /></figure>

minimal variance

<figure><img src="image_14.png" /></figure>

maximal variance

Sadly not my little sister

<figure><img src="image_15.png" /></figure>
# Data preprocessing

Per capita cheese consumption
correlates with

Number of people who died by becoming tangled in their bedsheets

Correlation: 94.71% (r=0.947091)

<figure><img src="image_16.png" /></figure>

tylervigen.com

Data sources: U.S. Department of Agriculture and Centers for Disease Control & Prevention

http://www.tylervigen.com/spurious-correlations

<figure><img src="image_17.png" /></figure>
# Data preprocessing

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /><figcaption>Calvin and Hobbs, Bill Watterson</figcaption></figure>

## Remove unrelevant variables

- Remove unrelated variables (spurious correlation)

* Crucial on datasets with many variables

- Remove a posteriori variables

* crash survival day7 = alcool+ crash speed + reeducation duration

## Remove irrelevant data

- Out of scope data (ex : holstein model, montbeliarde data)

- Outliers (PCA, boxplots, ...)

* Wrong data (ex broken sensor) ⇒ discard

* Rare events (ex mastitis) ⇒ kept or not?

<figure><img src="image_21.png" /></figure>
# Data preprocessing

* Data smoothing

* OK when we expect few changes within neighbors (time, space)

* Methods

* kernel methods (ex : rowing average)

* rowing median

* splines

* ...

<figure><img src="image_22.png" /></figure>
# Data preprocessing

<figure><img src="image_23.png" /></figure>

median filter, black 5 points, red 17 points

* Smoothing strength is a compromise.

* That depends on your question/model.

* An on your data variability.

<figure><img src="image_24.png" /></figure>
<figure><img src="image_25.png" /><figcaption>milky way, with increasing size gaussian kernel</figcaption></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>
<figure><img src="image_33.png" /><figcaption>milky way, with increasing size gaussian kernel</figcaption></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>
<figure><img src="image_42.png" /><figcaption>milky way, with increasing size gaussian kernel</figcaption></figure>

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>
<figure><img src="image_51.png" /><figcaption>milky way, with increasing size gaussian kernel</figcaption></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>
# Aggregate variables

## When?

* The same thing is measured multiple times

* Ex : captor1 and captor2

* Rare gold standard v.s. frequent low quality measurments

* Strong correlation between variables

* Ex behavioural traits

## How?

* Choose a variable

* Do a variable "mix" (ex linear combination)

* PCA : synthetic variables (a linear combination)

* Don't overinterpret PCA synthetic variables

<figure><img src="image_65.png" /></figure>
# Is my model OK or garbage?

<figure><img src="image_66.png" /></figure>

* A model can be anything

* Linear model http://cpc.cx/kQE, http://cpc.cx/kQF

* Non linear model http://cpc.cx/kQG

* A classifier http://cpc.cx/kQR

* ...

* The devil himself in a unicorn disguise

⇒ Garbage in, garbage out

* We assume that input data is ok, is the model garbage?

<figure><img src="image_67.png" /></figure>
Typical evil models : extrapolation

MY HOBBY: EXTRAPOLATING

<figure><img src="image_68.png" /></figure>

xkcd.com, Randall Munroe, CC-By-Cn2.5

* Your training and testing data must cover your practical use cases

<figure><img src="image_69.png" /></figure>
Typical evil models : Overfitting

<figure><img src="image_70.png" /></figure>

Underfitted

<figure><img src="image_71.png" /></figure>

Good Fit/Robust

<figure><img src="image_72.png" /></figure>

Overfitted

* Many parameters, few data ⇒ Check degree of freedom

<figure><img src="image_73.png" /></figure>
Typical evil models : outliers

<figure><img src="image_74.png" /></figure>

* Outlier with heavy weight ⇒ Check for outliers

* Check your question

* General model : keep Hulk, rename axis

* Dwarf model : discard Hulk

**WARNING :** evil models exists in unexpected cases
⇒ **ALWAYS TEST YOUR MODELS**

<figure><img src="image_75.png" /></figure>
# Test your models

<table><thead><tr><th></th><th colspan="2">detect mine</th><th colspan="2">detect rock</th></tr></thead><tbody><tr><th>mine</th><td>9</td><td>1</td><th>mine</th><td>5</td><td>5</td></tr><tr><th>rock</th><td>50</td><td>50</td><th>rock</th><td>10</td><td>90</td></tr></tbody></table>

Two mines detectors, what's the best one?

* All mistakes are NOT equivalent

* Be careful, some methods (like area under ROC curve) say so

* When possible use a real-life criterion (human lives, money, ...)

* When not, define a a reasonable criterion **apriori**.

<figure><img src="image_76.png" /></figure>
Test your models

* What the difference between model prediction and model output

* Plot observed v.s. predicted

* Model fitting (ex sum of square)

* Cross with other knowledge source (common sense, litterature, meta analysis ...)

* Model testing

* Use an **independant** dataset

* Test the model on it, according to your criterion

* See bootstrap like methods.

<figure><img src="image_77.png" /></figure>
Conclusion

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>