Evaluation of the Nutri-plus Module
Phase: NA
Conditions: Dietary Assessment
Sponsor: TNO

 
 
Study protocol 
 
 
 
Evaluation of the Nutri-plus module 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Name project lead: Dide Reijmer 
 
Project number: 060.51589/01.01 
Client: TNO  
 
Version and date: version 1, 11 May 2022 
 
 
 
 
 

 
 
Contents 
 
Evaluation of the Nutri-plus module....................................................................................................... 1 
1 
Introduction ................................................................................................................................ 3 
2 
Goal ............................................................................................................................................. 5 
3 
Study design ................................................................................................................................ 5 
4 
Study population ......................................................................................................................... 6 
5 
Description of the study .............................................................................................................. 7 
6 
Outcome variables ...................................................................................................................... 9 
7 
Description of incidents, safety, adverse events, side effects .................................................... 9 
8 
Data-analysis and statistics ......................................................................................................... 9 
9 
Risk analysis relating to the conduct of the study .................................................................... 10 
10 References ................................................................................................................................ 10 
 
 
 
 

 
 
1 
Introduction 
The Nutri+ module is part of the TNO 360-degree diagnostics tool (Harakeh et al., submitted). The aim 
of this tool is to generate a holistic perspective on an individual’s health; that is, in addition to biomedical 
health, it also maps an individual’s behavior, mental health, and socio-economic environment. This tool 
can be used for decision support and shared decision-making in, for example, primary care (Kalache 
et al., 2019). Furthermore, based on this more holistic diagnostic approach, more personalized advice 
or better-tailored interventions can be offered to individuals. 
However, the dietary questionnaire that is currently part of the 360-degree diagnostics tool is still fairly 
limited; it assesses consumption in only four food groups. In contrast, the Health Council distinguishes 
15 product groups (Gezondheidsraad, 2015). As a result, the current 360-degree diagnostics tool 
provides an insufficient picture of an individual’s dietary intake. In order to better assess dietary behavior 
within the 360-degree diagnostics and to provide more appropriate advice, the dietary questionnaire 
was further developed last year into the so-called “Nutri+ module.” 
The new questionnaire consists of 38 questions on an individual’s dietary intake across 17 product 
groups. The questionnaire and cutoff values are based on the Guidelines for Healthy Diet (Kromhout et 
al., 2016), where applicable further refined using the recommendations of the Netherlands Nutrition 
Centre. After completing the questionnaire, a profile wheel (see Figure 1) is displayed, providing insight 
into compliance with the guidelines for 17 product groups. In this profile wheel, a “traffic light” color is 
used for each product group to indicate whether an individual roughly meets the guidelines (green), 
consumes more than 50% of the recommended amount (orange), or consumes less than 50% of the 
recommended amount (red). For (sub)product groups for which no guideline exists, a grey color is 
shown (for example, for potatoes). 
 
 
Figure 1. Example of a “Profile wheel” for an individual after filling out the Nutri-plus module. 

 
 
The product groups defined by the Health Council were further refined to provide participants with more 
detailed insight into areas of their dietary pattern that could be improved. In addition, unhealthy choices 
are included in the Nutri+ module, even though they are not part of the Guidelines for Healthy Diet 
(Gezondheidsraad, 2015). By assessing and displaying these in the profile wheel, individuals can also 
gain insight into the extent of their unhealthy choices. 
With the Nutri+ module, we aim to offer a dietary intake assessment as an integral part of the 360-
degree tool, enabling individuals to gain insight into their own dietary patterns and areas for 
improvement. These outcomes can be used to provide personalized dietary advice or interventions 
tailored to the individual. 
Before the Nutri+ module can be used for personalized dietary assessment and advice, it must be 
evaluated to determine whether it provides an accurate picture of an individual’s dietary intake. To this 
end, the outcomes of the Nutri+ module, consisting of a red–orange–green categorization, can be 
compared with those of an already validated method for dietary intake assessment. In the literature, 
(weighed) dietary recall methods are commonly used to validate dietary questionnaires (Cade, 2002), 
as these methods are less sensitive to underreporting than food frequency questionnaires (Prentice, 
2011). 
In a 24-hour recall method, individuals are asked to provide detailed information about all foods and 
beverages consumed in the previous 24 hours (Meijboom, 2017). To obtain a reliable estimate of 
habitual dietary intake, the 24-hour recall must be repeated on multiple days in order to correct for day-
to-day variation. Wageningen University & Research has developed and validated a Dutch online 24-
hour dietary recall method that can be completed entirely online by participants, known as “Compl-Eat” 
(Meijboom, 2017). This significantly reduces the costs and burden for both researchers and participants. 
This study will consist of two phases. The first phase will involve a pre-test to assess the usability, 
comprehensibility, and completeness of the developed Nutri+ module among end users and experts. 
The insights generated will be used to further improve the Nutri+ module and the profile wheel 
visualization. After completion of the first phase, the second phase will commence, in which the Nutri+ 
module will be compared with a reference method in an evaluation study. The selected reference 
method for this evaluation study is a 24-hour recall using Compl-Eat. 
  
 

 
 
2 
Goal 
Goal of the Nutri+ module: To assess, at the individual level, the extent to which a person complies 
with the dietary guidelines and where there is room for improvement. These insights are used to provide 
personalized dietary advice and to monitor an individual’s dietary intake. 
Phase 1: Pre-test 
Objectives 
 To gain insight into the user experiences of the target group with completing the questionnaire 
and with regard to the visualization of the profile wheel. 
 To collect feedback from experts on the Nutri+ module (including completeness, question 
formulation, example products, etc.). 
 To use the collected user insights and expert feedback to further improve the Nutri+ module 
and the profile wheel visualization before starting the second phase of the study (the 
evaluation study). 
Phase 2: Evaluation study 
Main objective: To evaluate whether the absolute intake in grams, as measured by the Nutri+ 
module, is comparable to Compl-Eat for the 21 product groups. This will be explored both 
quantitatively (e.g., what percentage of intake is covered by the Nutri+ module compared with Compl-
Eat) and qualitatively (e.g., where the differences between Compl-Eat and the Nutri+ module lie). 
These insights can be used to further improve the tool. 
Sub-objectives: 
 To investigate whether the results of the Nutri+ module are reproducible over time or whether 
time effects are present, with a correlation coefficient of ≥ 0.75 between two measurement 
points considered successful. 
 To evaluate the Nutri+ module (dietary questionnaire) against the Compl-Eat 24-hour recall 
method. Specifically, we aim to examine whether the Nutri+ module can accurately classify at 
least 75% of individuals into three categories for the 21 product groups. 
 To evaluate the visualization of the profile wheel by users of the Nutri+ module. 
 
3 
Study design 
This is a longitudinal randomized study to evaluate the Nutri+ dietary intake module. The Nutri+ module 
can be completed by participants at home via an online portal; it is filled out only once and takes 
approximately 30 minutes to complete. As a reference method, Compl-Eat is used. This is an online 
24-hour recall method developed by Wageningen University & Research (Meijboom et al., 2017). 
Participants in the study can complete this 24-hour recall themselves at home via a portal, recording 
everything they ate and drank in the previous 24 hours. Based on the submitted data, the Compl-Eat 
platform can determine for an individual how much and what was consumed at both the product group 
and individual product levels. Compl-Eat is completed by participants on four non-consecutive days—
three weekdays and one weekend day (Saturday or Sunday)—to obtain an accurate representation of 
an individual’s average intake. The 24-hour recalls are manually checked for completeness by a 
researcher afterwards. 
A randomized cross-over design is used, in which one group of participants completes the Nutri+ 
module first, while the other group completes Compl-Eat first. The days on which participants complete 

 
 
the Nutri+ questionnaire and the days on which Compl-Eat is completed are randomly selected. There 
is a minimum interval of 7 days between the Nutri+ module and the first or last day of Compl-Eat. 
In addition, possible time effects in dietary recording and the reproducibility of the Nutri+ module results 
are examined. To this end, participants are invited again after three to six months to repeat the entire 
procedure of completing the Nutri+ module once and Compl-Eat four times. The first series of 
questionnaires is sent out in April; the second series begins in September. 
See Figure 2 for a schematic overview of the study design. 
Figure 2. Schematic overview of the study design of the evaluation study. 
 
4 
Study population 
A total of 200 participants will be recruited for this study. The target population consists of healthy men 
and women aged 35 to 80 years with normal weight, overweight, or obesity. 
Inclusion criteria for this study are: 
• 
BMI ≥ 20 and < 40 kg/m². 
• 
Age 35–80 years. 
• 
Good command of the Dutch language. 
• 
Reasonable digital literacy and access to a PC or laptop. 
• 
A fairly typical Dutch dietary pattern. 
Exclusion criteria for this study are: 

 
 
• 
Pregnant women or women who are breastfeeding. 
• 
Recent unintentional weight loss of more than 5 kg. 
• 
Under treatment for a serious (chronic) illness. 
• 
History of myocardial infarction or stroke in the past 5 years. 
• 
Previous or planned bariatric surgery. 
In this study, we aim to perform subgroup analyses to examine whether there are differences in the 
usability of the Nutri+ module between men and women, across different age groups, between low and 
high socioeconomic status, and between individuals with normal weight and those with overweight or 
obesity. The rationale for this is the known differences in dietary intake among these subgroups and 
the potentially higher risk of over- or underreporting. Therefore, during recruitment, we will strive for a 
proportional distribution of sex (male/female), age (35–55 / 55–80), educational level 
(primary/secondary vocational or higher education/university), and BMI (20–25 / 25–30 / 30–40) in the 
study population. For sex, this will already be considered during screening: once 100 participants of a 
particular sex are recruited, only participants of the other sex will be admitted. 
Recruitment will mainly take place via social media platforms (Facebook, LinkedIn, Instagram). If 
necessary, recruitment will be expanded through other channels, such as local newsletters and 
websites (e.g., “In de Buurt”). The number of participants represents our minimum target for data 
collection to ensure valid and reliable outcomes and to allow for subgroup analyses. Recruitment will 
therefore continue until the minimum number of participants is reached; additional participants are 
welcome (up to a maximum of 300). There will be no reserve participants. 
5 
Description of the study  
First Phase 
The first phase consists of preparatory activities, including the development of the Nutri+ module. 
During this phase, a pre-test is conducted with both participants and experts, which can take place 
either online or in person at a TNO location, depending on whether a visit to TNO is feasible. The aim 
of this pre-test is to assess the usability and comprehensibility of the Nutri+ module through a think-
aloud protocol supervised by a researcher. This method involves asking participants and experts to 
verbalize all their thoughts, feelings, and opinions while performing a task. This provides the researcher 
with deeper insights into the user experience, the interpretation of the questions, the respondent’s 
thought process, as well as the completeness of the questionnaire, the alignment of response options 
with the questions, and the overall clarity and understandability of the questionnaire. 
The think-aloud protocol is preferably conducted in a (digital) environment where the end 
user/respondent would normally complete the questionnaire; if this is not possible, it can also be carried 
out in a secluded, neutral room at TNO. Before starting, the researcher provides the participant or expert 
with an introduction to the study and requests consent for audio recording. The researcher remains 
passive while the participant or expert completes the questionnaire, but may remind them to continue 
verbalizing their thoughts if they fall silent. After completing the think-aloud protocol, several additional 
questions are asked regarding the experience of completing the questionnaire and the visualization of 
the profile wheel. 
Both the target population of the Nutri+ module and a number of experts are involved in the pre-test. In 
total, approximately 10 participants are aimed for. Among the target population, the goal is to include a 
group with diverse characteristics in terms of age, sex, and educational level. Based on the results of 
this pre-test, the Nutri+ module will be further refined to prepare it for use in Phase 2. 
Table 1: Overview timeline and investment for participants pre-test  

 
 
 
How often and when 
Total time investment 
Go through the think-
aloud protocol 
1 time filling out Nutri+ module  
(spring 2022) 
1 time 30 minutes 
Answering additional 
questions 
1 time directly after filling out Nutri+ 
module (spring 2022) 
1 time 15 minutes  
 
Second Phase 
The second phase involves recruiting participants through social media campaigns and the completion 
of the Nutri+ module and Compl-Eat. Participants are recruited via social media advertisements, which 
link them to an online information sheet. After reading the information sheet, participants can provide 
online informed consent for participation in the study. They are then automatically directed to an online 
screening questionnaire (see inclusion and exclusion criteria). This screening questionnaire 
automatically assesses whether the individual meets the study’s selection criteria. Candidates who do 
not meet the criteria are informed immediately. Candidates who do meet the criteria are directed to a 
baseline questionnaire to collect various baseline data (medication use, physical activity, adherence to 
any dietary or nutritional rules, and allergies or intolerances). Participants are then informed about the 
further study schedule and procedures. 
The study uses a cross-over design, in which participants are randomly assigned to one of two 
sequences: starting with the Nutri+ module or starting with Compl-Eat. Participants who start with the 
Nutri+ module during the first series (autumn 2022) will start with Compl-Eat during the second series 
(spring 2023), and vice versa. 
For all participants, the days on which the Nutri+ module and Compl-Eat must be completed are 
randomly selected. Both the Nutri+ module and Compl-Eat are completed at home using login 
credentials for the respective online platforms. Participants are not informed of the full schedule in 
advance to minimize potential influence on their dietary behavior. One day before a participant’s first 
“assessment day,” they are contacted by email by a researcher, requesting that they complete either 
the Nutri+ module or Compl-Eat the following day. Participants can respond to this email if the 
scheduled day does not fit their personal planning. In that case, the researcher reschedules the 
assessment day, and the participant is contacted again at a later time. 
During the first series (autumn 2022), all participants complete Compl-Eat on four days within a two-
week period and complete the Nutri+ module once. A minimum interval of seven days is maintained 
between completing Compl-Eat and the Nutri+ module to prevent interference between the methods. 
After completing the first series (autumn 2022), participants are reminded that they will be contacted 
again in spring 2023 to complete the Nutri+ module and Compl-Eat. The same procedures are followed 
in spring 2023 as in autumn 2022. 
Participants are also asked to complete an evaluation questionnaire regarding their experiences with 
completing the questionnaire and the visualization of the dietary profile wheel. 
After data collection for the evaluation study is completed, the data are analyzed. Based on this 
analysis, recommendations will be made for adjusting the Nutri+ module (questionnaire, algorithms, 
and cut-off values) and for improving the visualization of the profile wheel. 
Tabel 2: Overview timeline and investment for participants evaluation study  
 
How often and when 
Total time investment 

 
 
Nutri+ module 
1 time filling out autumn 2022 
1 time filling out spring 2023 
 
2 times, 20 minutes. 40 
minutes in total. 
Compl-eat 
1 time 4 days filling out autumn 2022 
1 time 4 days filling out spring 2023 
 
80 minutes per time. 160 
minutes in total.  
Evaluation 
questionnaire  
1 time filling out 
15 minutes 
 
 
6 
Outcome variables  
To evaluate the Nutri+ module in comparison with Compl-Eat, it is first necessary to assess the extent 
to which the food groups used in the Nutri+ module are also available in Compl-Eat. The outcomes 
listed below depend on the results of this mapping. In addition, Compl-Eat will need to apply the same 
food group classification based on the same cut-off values per product group as used in the Nutri+ 
module. 
Primary outcomes 
 Intake in grams for the 17 product groups that are quantitatively assessed (Fruit, Vegetables, 
Legumes, Nuts and Seeds, Fish, Dairy, Meat, Eggs, Soft Drinks, Alcohol, Coffee, Tea, 
Snacks, Sweet Spreads, Chips/Nuts/Party Snacks, Fast Food, Desserts/Ice Cream/Cakes) 
for both Compl-Eat and the Nutri+ module. 
Secondary outcomes 
 Evaluation overall and by subgroup (sex, BMI, etc.) for all product groups available in both 
Compl-Eat and the Nutri+ module, for which guidelines exist to assess reproducibility and 
classification. For Nutri+, these include: Fruit, Vegetables, Butter and Cooking/Frying 
Products, Dressings and Oils, Legumes, Nuts and Seeds, Fish, Dairy, Meat, Eggs, Cereal 
Products, Potatoes, Soft Drinks, Alcohol, Coffee, Tea, Snacks, Sweet Spreads, 
Chips/Nuts/Party Snacks, Fast Food, Desserts/Ice Cream/Cakes. 
 Energy intake based on Compl-Eat 
 Expected energy intake based on sex, age, weight, and physical activity 
 Evaluation of the profile wheel 
 
7 
Description of incidents, safety, adverse events, side effects   
The study for participants consists of completing dietary questionnaires and 24-hour dietary recalls. 
Due to the nature of the study, no incidents or adverse events are expected. Any complaints regarding 
the conduct of the study, the websites used, or other questions or remarks can be submitted via email 
or telephone, as indicated in the participant information. 
 
8 
Data-analysis and statistics  
For Compl-Eat, all analyses will be based on the average intake per product group across the four 
assessment days within a single measurement series (spring or autumn). 

 
 
Based on the Compl-Eat data, it will be assessed whether there is excessive under- or over-reporting 
in the study population. To this end, the number of calories consumed, as calculated by Compl-Eat, will 
be compared with the expected caloric intake based on participants’ age, sex, height, and weight. 
The agreement between classifications in Compl-Eat and the Nutri+ module will be evaluated by 
assessing the percentage overlap in categorization and subsequently examining correlations between 
the scores. The goal is for 75% agreement per product group between the Nutri+ module and Compl-
Eat. If lower agreement is observed, the data will be qualitatively examined to identify areas for 
improvement in the Nutri+ module. 
In addition, the intake in grams per product group as measured by the Nutri+ module and Compl-Eat 
will be examined. Correlation coefficients will be calculated to assess agreement between the Nutri+ 
module and Compl-Eat. These outcomes are primarily exploratory and aim to provide insight into where 
the largest differences between Compl-Eat and Nutri+ occur. These insights can be used to further 
improve the Nutri+ module. 
To assess the reproducibility of the Nutri+ module, correlations between the spring and autumn 
measurements will be used. 
Participants’ opinions on the visualization of the profile wheel will be evaluated using descriptive 
information. 
 
9 
Risk analysis relating to the conduct of the study   
Given the nature of the study, no health risks are expected (see Section 7). A potential risk during the 
study is insufficient participant recruitment. In that case, recruitment methods can be expanded from 
social media only to other channels, such as newsletters, local newspapers, and similar. However, 
based on previous positive experiences with social media recruitment, this is not expected to be 
necessary. Another risk is that the distribution of sex, age, and BMI may be insufficient to perform 
subpopulation analyses. Sex will be accounted for during screening; in other words, once 100 men have 
been recruited, only women will be admitted to the study. Unfortunately, it is not feasible to automate 
screening for multiple criteria. For other subgroups (age, BMI, and SES), a targeted recruitment 
campaign may be employed if the distribution is skewed. In the worst-case scenario, subpopulation 
analyses may not be feasible. 
 
10 
References 
Cade, J., Thompson, R., Burley, V., & Warm, D. (2002). Development, validation and utilisation of food-
frequency questionnaires–a review. Public health nutrition, 5(4), 567-587.  
Gezondheidsraad. Richtlijnen goede voeding 2015. Den Haag: Gezondheidsraad, 2015; publicatienr. 
2015/24. 
Harakeh H. et al. 360° Diagnostic Tool to Personalize Lifestyle Advice in Primary Care for People with 
Type 2 Diabetes: Development and Usability Study. JMIR Formative Research (submitted).  
Kalache, A., de Hoogh, A. I., Howlett, S. E., Kennedy, B., Eggersdorfer, M., Marsman, D. S., ... & 
Griffiths, J. C. (2019). Nutrition interventions for healthy ageing across the lifespan: a conference report. 
European journal of nutrition, 58(1), 1-11. 

 
 
Kromhout D, Spaaij CJ, de Goede J, Weggemans RM. The 2015 Dutch food-based dietary guidelines. 
Eur J Clin Nutr. 2016 Aug;70(8):869-78. doi: 10.1038/ejcn.2016.52. Epub 2016 Apr 6. PMID: 27049034; 
PMCID: PMC5399142. 
Meijboom, S., van Houts-Streppel, M. T., Perenboom, C., Siebelink, E., van de Wiel, A. M., Geelen, A., 
Feskens, E., & de Vries, J. (2017). Evaluation of dietary intake assessed by the Dutch self-administered 
web-based dietary 24-h recall tool (Compl-eat™) against interviewer-administered telephone-based 24-
h recalls. Journal of nutritional science, 6, e49. https://doi.org/10.1017/jns.2017.45 
Prentice, RL, Mossavar-Rahmani, Y, Huang, Y, et al. (2011) Evaluation and comparison of food 
records, recalls, and frequencies for energy and protein assessment by using recovery biomarkers. Am 
J Epidemiol 174, 591–603.