## 4 Methodology
### Data
#### 4.5.2 Data Preprocessing  
The data have gone through several changes over time. As an illustration, there were 293 variables in the ENEM questionnaire in 2009, while in the following year, 2010, only 57. Added to this difference in the number of variables collected, and there were also changes related to the variables representation, such as: (1) features were binary for some years and quantity for others; (2) categories were represented by numbers in some years and by strings in others; and (3) categorical features with \(i\) options were transformed into \(i\) binary features for different years. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **ENEM—BEFORE** & \multicolumn{4}{c}{**FINAL DATASET—AFTER**} \\ \hline
**YEAR** & School & Student & School * & Student & Teacher \\ \hline
**2009** & 22,696 & 1,173,419 & 15,413 & 631,604 & 492,584 \\
**2010** & 22,496 & 1,096,483 & 17,957 & 774,937 & 553,909 \\
**2011** & 22,274 & 1,200,923 & 19,584 & 911,309 & 602,292 \\
**2012** & 22,240 & 1,201,036 & 20,120 & 954,009 & 614,749 \\
**2013** & 21,085 & 1,293,786 & 21,085 & 1,049,134 & 649,652 \\
**2014** & 22,846 & 1,344,736 & 21,383 & 1,070,778 & 671,383 \\
**2015** & 21,843 & 1,310,702 & 21,843 & 1,128,398 & 701,424 \\
**2016** & 24,217 & 1,523,161 & 22,476 & 1,192,812 & 709,619 \\
**2017** & 24,102 & 1,433,841 & 21,692 & 1,078,598 & 676,138 \\
**2018** & 22,729 & 1,151,207 & 20,617 & 920,310 & 615,011 \\
**2019** & 21,069 & 933,988 & 18,149 & 777,589 & 586,718 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of schools, students before and after scope definition and the number of teachers used to build teacher variables in the school level. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \(T_{start}\) & **2009** & **2010** & **2011** & **2012** & **2013** & **2014** & **2015** & **2016** & **2017** & **2018** \\ \hline NUMBER OF SCHOOLS & 7.988 & 9.357 & 10.013 & 10.410 & 10.829 & 11.131 & 11.448 & 11.744 & 11.985 & 12.561 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of schools present in each time window. It is important to standardize the data to overcome these issues and to allow us to compare the findings over the years. First, we select only variables present in all waves. Next, the data were standardized regarding content and meaning. A variable with less information was used as a reference for mapping the others. For instance, if a variable was binary in one year and multiple categorical in the others, the binary version was adopted for all years. The income features were normalized using a contemporary minimum wage. The variable related to the use of technological devices were individually treated. For example, before 2019, the available information on technology devices at school was measured by just one variable (student's computer), whereas in 2019, the questions also asked about notebooks and tablets, which were unified. The missing values were analyzed separately since there were not many and were used as the most probable value. Alternatively, the mean was used for those that did not have a clear explanation. To reduce the influence of outliers, all numerical features were normalized for each year separately, using the \(\alpha\)-winsorized values of the distribution (\(\alpha/2=0.025\) at each tail) as their minimum and maximum. As a strategy to enhance the discriminant power of data, some variables frequently brought to the fore in discussions on the quality of secondary education (39), which were not initially present in the databases, were created.