TITLE: FSQC Protocol
AUTHORS: Saashi Bedford

[DESCRIPTION]
This protocol is a guide to conducting a visual quality control (QC) of Freesurfer outputs. Quality control is important to make sure that the cortical surface reconstruction and grey and white matter classification are accurate.

To examine outputs, ten images per subject are generated (see https://github.com/sbedford0/Generate_FSQC_images), showing the cortical segmentation and surface boundaries at different slices and views of the brain (3 axial; 3 coronal; 4 sagittal). The pial surface is delineated in red, and the white matter surface in blue. These images should be examined to check that the reconstructed surfaces follow the cortical boundaries accurately. 

[GUIDELINES]
Rating criteria:

BAD
Missing large area, or multiple small/medium areas, of cortex
Large errors in grey/white matter boundary

MINOR ERROR 
Missing small area of cortex
Minor over or under segmentation
Misclassification of GM/WM in one or two regions

GOOD
Accurate delineation of cortical boundary across the brain
No cortex missing

MOTION / POOR SCAN QUALITY
Visible motion artefact or poor overall scan quality (i.e. ringing, blurring)

Note: If Freesurfer surface looks ok or has minor errors, but scan quality is poor, rate as MOTION. If there is visible motion in the scan and Freesurfer has also performed badly (large or significant errors), rate as BAD


[STEPS]
SECTION: Instructions
1. Download, save and unzip the Image Rating QC app from https://github.com/sbedford0/imageratingQCApp
SECTION: Instructions
2. Follow the instructions in the “readme” file to run the program:
The first time you run the program, in a terminal, run ‘npm install’ in the app directory
Run ‘npm run build’ to build the app for production in the build folder and create a local version (you only need to do this once)
Once you have done this, any time you want to start the app, run ‘npm run start’ in the app directory to launch the app in a web browser
Click ‘browse’ to select and load in your images and start rating! 
Note: if you have a large dataset, try to limit yourself to maximum ~2000 subjects in one session to avoid fatigue and distraction
SECTION: Instructions
3. Images will be displayed one by one
SECTION: Instructions
4. Rate by either clicking the buttons or using arrow keys 
Thumbs down (← key):       BAD
Shrug (↓ key):                     MINOR ERROR 
Thumbs up (→ key):            GOOD
Brain with ripples (↑ key):  VISIBLE MOTION / POOR SCAN QUALITY

See Guidelines for rating criteria, and Materials for examples of outputs and ratings
SECTION: Instructions
5. When you have gone through all of the images, a download button will appear. Click this to download your ratings as a csv file.
SECTION: Instructions
6. After downloading your ratings, convert scores to numeric
GOOD = 1
MINOR ERROR = 2
MOTION = 3
BAD = 4
SECTION: Instructions
7. Calculate an average to generate a score between 1-4 for each subject
SECTION: Instructions
8. Ideally have (at least) one other rater score your images 
Compare scores (average score per participant), and take an average and/or resolve any major discrepancies between scores
Check correlation and/or ICC between raters to make sure inter-rater reliability is reasonable
SECTION: Instructions
9. Apply your exclusion criteria to exclude scans with a score below a certain threshold or cut off, or use these scores as a covariate in your analysis