Methods and systems using fractional rank precision and mean average precision as test-retest reliability measures

Disclosed herein are methods and systems of evaluating test-retest precision using fractional rank precision or mean-average precision, comprising: a) collecting a test and a retest result of each subject, wherein the results are described in feature space(s) and collected from a vision test machine; b) selecting, a first test result of a first subject; c) calculating distances from the first test result to the retest result of each subject; d) assessing, a similarity between the first test result and the retest result of each subject by ranking the distances in a non-descending order; e) assessing a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) repeating b), c), d), and e) for each subject; and evaluating, the test-retest precision based on the rank precision for each of the plurality of subjects.

BACKGROUND

Clinically test-retest is typically used in monitoring of disease progression and/or effects of therapeutic treatment. Test—retest reliability is indicated by the variation in measurement results taken on a same subject in a test and in a subsequent retest. Such variability can be caused by a variety of factors including intra-individual variability and/or variability in measurement devices. A measurement may be considered as reliable when this variation is smaller than a pre-determined acceptance threshold.

SUMMARY OF THE INVENTION

Vision is one of the most important senses for many everyday tasks, and blindness ranks highly among most-feared ailments. Therefore, early diagnosis and treatment of vision loss are critical. As a result, regular monitoring of visual function at least in at-risk populations would be desirable. Certain vision tests rest on computationally intensive algorithms to provide a more comprehensive description of visual function. Thus, they have the potential to improve clinical care and research by more precise measurement of the effects of disease progression or ophthalmic interventions. A common proxy to study precision of a vision test is to assess its test-retest reliability; however, standard methods to assess clinical precision and test-retest reliability may be inadequate or even misleading for the more complex, higher-dimensional test outputs. Two major sources of imprecision typically are noise in the measurement device and moment-to-moment variability of the physiological phenomenon under observation. To estimate the contribution of noise in the device, precision is often assessed by the reliability of repeated measurements. The standard tools for this assessment are the intra-class correlation coefficient (ICC) and the Bland-Altman coefficient of repeatability (CoR). However, it is important to note that repeatability measures are an indirect assessment only; in the extreme case, a test with a binary outcome, as a non-limiting example, light perception, have almost perfect reliability but little discriminatory power for most of the population. Furthermore, the ICC is dominated by the values at either end of the test range and is therefore sensitive to outliers. The Bland-Altman coefficient of repeatability, which is defined as 1.96 times the standard deviation of differences between repeated measurements, does not suffer from this problem, and also may provide an intuitive threshold for how much change between two tests should be considered statistically significant. However, this threshold rests on the assumption that tests are homoscedastic, as a non-limiting example, the measurement error is independent of the magnitude of the ground truth (e.g. patients with poor vision perform tests as reliably as normal-sighted controls); its usefulness is limited by quantization of many tests, in addition to the evidence of heteroscedasticity in vision testing data. Moreover, absolute CoR values do not directly relate to clinical meaningfulness, and do not allow comparing the reliability of different tests with outputs of different magnitude or different dimensionality.

The methods and systems disclosed herein, in various embodiments, adapt at least one method in machine learning and/or information retrieval field and apply it in the measurement of similarity in test-retest pairs of at least one subject. In some embodiments, the methods and systems disclosed herein include at least a mean average precision method (MAP) adapted from machine learning and/or information retrieval. In some embodiments, the methods and systems disclosed herein include at least a fractional ranking precision (FRP) method derived from machine learning and/or information retrieval field. The methods and systems disclosed herein, for non-limiting examples, MAP and FRP, improve the limitations of traditional methods for the measurement of test-retest reliability and precision. Additionally, the methods and systems disclosed herein provide sensitivity (detection of subtle changes) and robustness (in the presence of artifacts) in evaluating the effectiveness of vision-based features in the detection of critical vision changes caused by disease progression and/or therapeutic interventions. Furthermore, the methods and systems disclosed herein enable assessment of vision-based features in multi-dimensional feature space.

In one aspect, disclosed herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a retest result of each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) selecting, by the computer, a first subject from the plurality of subjects and a first test result of the first subject; c) calculating, by the computer, distances from the first test result to the retest result of each of the plurality of subjects; d) assessing, by the computer, a similarity between the first test result and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) repeating, by the computer, steps b), c), d), and e) for each of the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a quick CSF (qCSF) testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a retest result from each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In another aspect, disclosed herein are non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application for evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a database, in a computer memory, of a test result and a retest result for each of a plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces, and one of the test result and the retest result is obtained from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In another aspect, disclosed herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) selecting, by the computer, a first subject from the plurality of subjects and a first test result of the first subject; c) calculating, by the computer, distances from the first test result to each of the plurality of retest results of each of the plurality of subjects; d) assessing, by the computer, a similarity between the first test result and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the first subject based on the rank of distances from the first test result to each of the plurality of retest results of the first subject; f) repeating, by the computer, steps b), c), d), and e) for each of the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to each of the plurality of retest results of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on the rank of distances from the first test result to each of the plurality of retest results of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In yet another aspect, disclosed herein are non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application for evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a database, in a computer memory, of a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to each of the plurality of retest results of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on the rank of distances from the first test result to each of the plurality of retest results of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In yet another aspect, disclose herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a retest result of each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) selecting, by the computer, a nth subject from the plurality of subjects and the test result of the nth subject; c) calculating, by the computer, distances from the test result of the nth subject to the retest result of each of the plurality of subjects; d) assessing, by the computer, a similarity between the test result of the nth subject and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the nth subject based on the rank of the distance from the test result of the nth subject to the retest result of the nth subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In yet another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a retest result from each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) a software module configured to select a nth subject from the plurality of subjects and a test result of the nth subject; c) a software module configured to calculate distances from the test result of the nth subject to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the test result of the nth subject and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the nth subject based on a rank of a distance from the test result of the nth subject to the retest result of the nth subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In yet another aspect, disclosed herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) selecting, by the computer, a nth subject from the plurality of subjects and a test result of the nth subject; c) calculating, by the computer, distances from the test result of the nth subject to each of the plurality of retest results of each of the plurality of subjects; d) assessing, by the computer, a similarity between the test result of the nth subject and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the first subject based on the rank of distances from the test result of the nth subject to each of the plurality of retest results of the first subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In yet another aspect disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a retest result from each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) a software module configured to select a nth subject from the plurality of subjects and a test result of the nth subject; c) a software module configured to calculate distances from the test result of the nth subject to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the test result of the nth subject and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the nth subject based on a rank of a distance from the test result of the nth subject to the retest result of the nth subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In yet another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) a software module configured to select a nth subject from the plurality of subjects and a test result of the nth subject; c) a software module configured to calculate distances from the test result of the nth subject to each of the plurality of retest results of each of the plurality of subjects; d) a software module configured to assess a similarity between the test result of the nth subject and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the nth subject based on the rank of distances from the test result of the nth subject to each of the plurality of retest results of the nth subject; f) a software module configured to repeat b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

DETAILED DESCRIPTION OF THE INVENTION

The method and systems disclosed herein, in various embodiments, adapt at least one method in machine learning and/or information retrieval field and applies it in the measurement of similarity in test-retest pairs of at least one subject. In some embodiments, the methods and systems disclosed herein include at least a mean average precision method (MAP) adapted from machine learning and/or information retrieval. In some embodiments, the methods and systems disclosed herein include at least a fractional ranking precision (FRP) method derived from machine learning and/or information retrieval field. The methods and systems disclosed herein, for non-limiting examples, MAP and FRP, improve the limitations of traditional methods for the measurement of test-retest reliability and precision. Additionally, the methods and systems disclosed herein provide sensitivity (detection of subtle changes) and robustness (in the presence of artifacts) in evaluating the effectiveness of vision-based features in the detection of critical vision changes caused by disease progression and/or therapeutic interventions. Furthermore, the methods and systems disclosed herein enable assessment of vision-based features in multi-dimensional feature space.

In one aspect, disclosed herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a retest result of each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) selecting, by the computer, a first subject from the plurality of subjects and a first test result of the first subject; c) calculating, by the computer, distances from the first test result to the retest result of each of the plurality of subjects; d) assessing, by the computer, a similarity between the first test result and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) repeating, by the computer, steps b), c), d), and e) for each of the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a retest result from each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In another aspect, disclosed herein are non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application for evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a database, in a computer memory, of a test result and a retest result for each of a plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces, and one of the test result and the retest result is obtained from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects. In some embodiments, the test is a first vision test. In some embodiments, the retest is the first vision test or a second vision test. In some embodiments, the first vision test or the second vision test is one or more selected from: a vision acuity test, a CSF test, and an OCT test. In some embodiments, the feature space is one-dimensional or multi-dimensional. In some embodiments, the feature space comprises a feature. In some embodiments, the feature includes one or more features selected from: a median AULCSF computed over the spatial frequency range of 1.5 to 6 cpd, a median AULCSF computed over the spatial frequency range of 6 to 12 cpd, a median AULCSF computed over the spatial frequency range of 12 to 18 cpd, a median AULCSF computed over the spatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity for at least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and a spatial frequency at which a CSF reaches a pre-determined contrast threshold. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of subjects in the plurality of subjects. In some embodiments, the rank is a real number ranging from 0 to N−1, N being a total number of retests of the plurality of subjects. In some embodiments, the distance is one or more selected from: a Euclidean distance, a Manhattan distance, and a Mahalanobis distance. In some embodiments, assessing the rank precision comprises: calculating a normalized rank, the normalized rank being the rank of the distance divided by a total number of subjects of the plurality of subjects; and calculating the rank precision, the rank precision being equal to one subtracted by the normalized rank. In some embodiments, the rank precision is an inverse of the rank of the retest result of the first subject. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of subjects. In some embodiments, the test-retest precision is mean of the rank precision for each of the plurality of retest. In some embodiments, the rank is a real number ranging from 1 to N, N being a total number of subjects of the plurality of subjects. In some embodiments, he rank is a real number ranging from 1 to N, N being a total number of retests of the plurality of subjects. In some embodiments, the vision test machine is one or more selected from: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, a OCT machine, a MRI machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling threshold machine, a tonometer machine, an aberrometer machine, an eye-tracking device, and an ocular alignment machine.

In another aspect, disclosed herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) selecting, by the computer, a first subject from the plurality of subjects and a first test result of the first subject; c) calculating, by the computer, distances from the first test result to each of the plurality of retest results of each of the plurality of subjects; d) assessing, by the computer, a similarity between the first test result and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the first subject based on the rank of distances from the first test result to each of the plurality of retest results of the first subject; f) repeating, by the computer, steps b), c), d), and e) for each of the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects.

In another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to each of the plurality of retest results of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on the rank of distances from the first test result to each of the plurality of retest results of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects.

In yet another aspect, disclosed herein are non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application for evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a database, in a computer memory, of a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) a software module configured to select a first subject from the plurality of subjects and a first test result of the first subject; c) a software module configured to calculate distances from the first test result to each of the plurality of retest results of each of the plurality of subjects; d) a software module configured to assess a similarity between the first test result and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the first subject based on the rank of distances from the first test result to each of the plurality of retest results of the first subject; f) a software module configured to repeat b), c), d), and e) for each of the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects.

In yet another aspect, disclose herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a retest result of each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) selecting, by the computer, a nth subject from the plurality of subjects and the test result of the nth subject; c) calculating, by the computer, distances from the test result of the nth subject to the retest result of each of the plurality of subjects; d) assessing, by the computer, a similarity between the test result of the nth subject and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the nth subject based on the rank of the distance from the test result of the nth subject to the retest result of the nth subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects.

In yet another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a retest result from each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) a software module configured to select a nth subject from the plurality of subjects and a test result of the nth subject; c) a software module configured to calculate distances from the test result of the nth subject to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the test result of the nth subject and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the nth subject based on a rank of a distance from the test result of the nth subject to the retest result of the nth subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects.

In yet another aspect, disclosed herein are computer-implemented methods of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) collecting, by a computer, a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) selecting, by the computer, a nth subject from the plurality of subjects and a test result of the nth subject; c) calculating, by the computer, distances from the test result of the nth subject to each of the plurality of retest results of each of the plurality of subjects; d) assessing, by the computer, a similarity between the test result of the nth subject and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) assessing, by the computer, a rank precision for the first subject based on the rank of distances from the test result of the nth subject to each of the plurality of retest results of the first subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) evaluating, by the computer, the test-retest precision based on the rank precision for each of the plurality of subjects.

In yet another aspect disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a retest result from each of the plurality of subjects, wherein the test result and the retest result are described in one or more feature spaces and one of the test result and the retest result is collected from a vision test machine; b) a software module configured to select a nth subject from the plurality of subjects and a test result of the nth subject; c) a software module configured to calculate distances from the test result of the nth subject to the retest result of each of the plurality of subjects; d) a software module configured to assess a similarity between the test result of the nth subject and the retest result of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the nth subject based on a rank of a distance from the test result of the nth subject to the retest result of the nth subject; f) repeating, by the computer, steps b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects.

In yet another aspect, disclosed herein are computer-implemented systems comprising a digital processing device comprising at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create an application of evaluating test-retest precision of a test and a retest of a plurality of subjects using fractional rank precision (FRP) or mean-average precision (MAP), comprising: a) a software module configured to collect a test result and a plurality of retest results from each of the plurality of subjects, wherein the test result and the plurality of retest results are described in one or more feature spaces, and one of the test result and the plurality of retest results is collected from a vision test machine; b) a software module configured to select a nth subject from the plurality of subjects and a test result of the nth subject; c) a software module configured to calculate distances from the test result of the nth subject to each of the plurality of retest results of each of the plurality of subjects; d) a software module configured to assess a similarity between the test result of the nth subject and the plurality of retest results of each of the plurality of subjects by ranking the distances in a non-descending order; e) a software module configured to assess a rank precision for the nth subject based on the rank of distances from the test result of the nth subject to each of the plurality of retest results of the nth subject; f) a software module configured to repeat b), c), d), and e) for 1≤n≤N, N being a total number of subjects in the plurality of subjects; and g) a software module configured to evaluate the test-retest precision based on the rank precision for each of the plurality of subjects.

Overview

Human vision is one of the most important senses for many everyday tasks, and blindness ranks highly among most-feared ailments. Therefore, early diagnosis and treatment of vision loss are critical. As a result, regular monitoring of visual function at least in at-risk populations would be desirable. Certain vision tests rest on computationally intensive algorithms to provide a more comprehensive description of visual function. Thus, they have the potential to improve clinical care and research by more precise measurement of the effects of disease progression or ophthalmic interventions. A common proxy to study precision of a test is to assess its test-retest reliability; however, standard methods to assess clinical precision and test-retest reliability may be inadequate or even misleading for the more complex, higher-dimensional test outputs. Two major sources of imprecision typically are noise in the measurement device and moment-to-moment variability of the physiological phenomenon under observation. To estimate the contribution of noise in the device, precision is often assessed by the reliability of repeated measurements. The standard tools for this assessment are the intra-class correlation coefficient (ICC) and the Bland-Altman coefficient of repeatability (CoR). However, it is important to note that repeatability measures are an indirect assessment only; in the extreme case, a test with a binary outcome, as a non-limiting example, light perception, may have almost perfect reliability but little discriminatory power for most of the population. Furthermore, the ICC is dominated by the values at either end of the test range and is therefore sensitive to outliers. The Bland-Altman coefficient of repeatability, which is defined as 1.96 times the standard deviation of differences between repeated measurements, does not suffer from this problem, and also may provide an intuitive threshold for how much change between two tests should be considered statistically significant. However, this threshold rests on the assumption that tests are homoscedastic, as a non-limiting example, the measurement error is independent of the magnitude of the ground truth (e.g. patients with poor vision perform tests as reliably as normal-sighted controls); its usefulness is limited by quantization of many tests, in addition to the evidence of heteroscedasticity in vision testing data. Moreover, absolute CoR values do not directly relate to clinical meaningfulness, and do not allow comparing the reliability of different tests with outputs of different magnitude or different dimensionality.

In some embodiments, the methods and systems disclosed herein relate to the information retrieval field. In some embodiments, the methods and systems disclosed herein use method(s) in information retrieval. The methods and systems disclosed herein, in various embodiments, are applied in order to evaluate similarity, repeatability, reliability, precision, or variability of a test-retest pair. In some embodiments, the methods and systems disclosed herein are used for multiple subjects with each subject having at least one test and at least one retest result. In some embodiments, method disclosed herein, for non-limiting examples, MAP and FRP, are particularly useful for test-retest results in higher-dimensional feature spaces including quick CSF (qCSF) results, traditional CSF measurements at different spatial frequencies, imaging modalities such as fundus photography or OCT, and perimetry results of different locations in the visual field. For traditional tests with scalar outcomes, it is relatively easy to statistically describe some aspects of test-retest variability, for example “95% of test-retest pairs will be less than X units apart.” However, there are limitations with that method (as a non-limiting example, dependence on test range and quantization); such limitations get much worse in a multi-dimensional space, when test-retest differences are expressed as a multi-dimensional vector. However, MAP or FRP methods are suited for describing relationships or point distances within higher-dimensional feature spaces. In some embodiments, MAP and FRP methods can not only be applied to individual features or combinations of features of a specific vision test, but also be applied to combine results from different tests including one or more functional, behavior, or structural tests. Further, MAP or FRP methods can be used to compare and select individual features or different combination of features based on their test-retest precision performance. In some embodiments, CoR and ICC are more vulnerable to artifacts than MAP and FRP for measuring test-retest similarity. In some embodiments, MAP and FRP are more accurate in capturing a test's precision by expressing test-retest variability in terms of population variability. In some embodiments, MAP and FRP penalize coarse quantization in the test-retest results.

Tests and Retests

In some embodiments, the methods and systems disclosed herein include a test and a retest. In some embodiments, a test and a retest is examination of one or more visionary feature using specialized and suitable medical machine capable of measuring the feature(s). In some embodiments, a test and a retest is examination of one or more visionary features as disclosed herein. In some embodiments, a test and a retest is a procedure whose result is described by one or more visionary features as disclosed herein. In some embodiments, a test and a retest is an examination or procedure using one or more specialized machine as disclosed herein. In some embodiment, a test or a retest is a same test with all testing conditions substantially identical for the test and the retest except the time of testing. In further embodiments, the testing conditions include one or more selected from: the testing subjects, the testing instruments, the testing procedure, the testing method, the result recording method, the testing time, the testing artifacts, and the testing noise level. In some cases, the testing conditions include one or more selected from: a test processing method, a test evaluation method and a test scoring method. In some cases, a test and a retest is performed with only one or more differences in testing conditions. As a nonlimiting example, the only substantial difference between testing conditions in a test and a retest may be the testing time. In some embodiments, a test or a retest is a same test with an identical testing protocol except the time of testing. In some cases, a test and a retest is performed with one or more differences in testing conditions. As a nonlimiting example, a test and a retest is performed before and after at least one therapeutic intervention, respectively, and also with different testing time. In some embodiments, a test and a retest is performed before and after at least some disease progression has taken place, respectively. In some embodiments, a test and a retest is a different test.

In some cases, a test or a retest as disclosed herein may be any suitable vision tests or medical tests that can be described in a feature space. As nonlimiting examples, a test or a retest may be one or more selected from: an acuity test, an acuity threshold test, an acuity slope test, a sensitivity test, a perimetry test, a color sensitivity test, a spatial frequency sensitivity test with at least one spatial frequency ranging from 0 cycles per degree of visual angle (cpd) to 50 cpd, a contrast sensitivity test, a contrast sensitivity function (CSF) test, a CSF test with at least one spatial frequency ranging from 0 to 50 cpd, a peak spatial frequency test, a peak sensitivity test, a sensitivity bandwidth test, an optical coherence tomography (OCT) test, a ultrasound test, a magnetic resonance imaging (MRI) test, an X-ray test, a microscopic test, a computerized adaptive contrast sensitivity test using a computerized contrast sensitivity testing device, a qCSF test using a qCSF testing device, or any other imaging modality. In some embodiments, a test or a retest examines at least one preselected parameter of at least a vision disease. In some embodiments, a test or a retest examines at least one preselected parameter of at least one therapeutic treatment or intervention.

In some cases, the test may be the first test being performed on the subject(s). Alternatively, the test may be a test preceding a number of retests. Alternatively, a test may be any test that proceeds or follows a certain number of retests. In some cases, each subject may have one or more retests. In further cases, the number of retests for each subject may or may not be identical. As a nonlimiting example, four human subjects may each have a test; three of the four subjects may have 3 retests each; the last subject may have 2 retests. The total number of subjects is 4, and the total number of retests may be 11. A test and a retest may be in the same feature space. Alternatively, a test and a retest may be in two different feature spaces.

In some cases, a test (retest) may not be limited to vision test. In some cases, a test (retest) may be any suitable test of physiological functions or pathological functions. In some cases, a test (retest) may be any suitable tests that may be described in a feature space.

Results

In some embodiments, at least one test and at least one retest is performed with a same subject. In some embodiments, a test and a retest includes at least one test result and a retest result. In some embodiments, a test and a retest includes at least a number of test results and a number of retest results, the number being any real number no smaller than 1. In some embodiments, a test result or a retest result is indicative of visual performance of a subject. In some embodiments, a test result or a retest result is indicative of at least one disease progression. In some embodiments, a test result or a retest result is indicative of effect of at least one therapeutic intervention.

In some embodiments, a test result or a retest result includes at least one test score. In some embodiments, a test (retest) score is a scalar. In further embodiments, a test (retest) result includes a score ranging anywhere from 0 to 1. In some embodiments, a test (retest) score is normalized by the maximum score or a pre-selected number to be within the range of 0 to 1. In some embodiments, a test (retest) score is any non-negative real number. In some embodiments, a test (retest) score is one or more real numbers. In some embodiments, a test (retest) score is multi-dimensional. In some embodiments, a test (retest) score is a vector. In some embodiments, a test (retest) score is a multi-dimensional vector. In some embodiments, the number of dimensions of a test (retest) score is determined by or equal to the number of dimensions of a feature space. In some embodiments, a test (retest) score includes at least one scalar score for each dimension in the feature space that a test and a retest was performed and/or measured.

In some embodiments, a test (retest) score in at least one dimension is weighted. In some embodiments, a test (retest) score or the raw data of a test or a retest is processed in at least one dimension. In some embodiments, the processing of a test (retest) score includes one or more selected from weighting, normalization, noise reduction, filtering, translation, quantization, and rounding up.

In some cases, a test or a retest as disclosed herein may be any suitable vision tests or medical tests that may be conducted on specialized medical machinery other than a generic computer. In some embodiments, a test (retest) result may be collected from a specialized machine that enables measurement of visionary parameters of human subjects. In some embodiments, the methods and systems, and digital processing devices as disclosed herein include a connection to one or more specialized machinery that enables measurement of visionary parameters of human subjects. In some embodiments, the specialized machinery disclosed herein may include but are not limited to: a computerized adaptive contrast sensitivity testing device, a qCSF testing device, an OCT machine, a Mill machine, a X-ray machine, an ultrasound machine, a visual field testing machine, a fundus photography system, a dark adaptation measurement machine, an auto-refractor machine, a frequency-doubling technology machine, a tonometer machine, an aberrometer machine, an eye-tracking device, an Adaptive Sensory Technology Sentio device (“Next-generation vision testing: the quick CSF” in Current Directions in Biomedical Engineering 2015; 1:131-134 by Don et. al.), and an ocular alignment machine. In some embodiments, the specialized vision test machine may not include a vision acuity chart. In some cases, there may be a wired or a wireless connection between the specialized machinery used to collect test (retest) results. In some embodiments, the wired or wireless connection may be direct. In other cases, the connection may be indirect, and the collected result(s) of raw data may be preprocessed at another machine using preprocessing steps as disclosed herein. In some embodiments, the methods and systems disclosed herein may include a processor or a digital processing device for preprocessing of raw data collected from a one or more specialized machines.

In some cases, the methods and systems as disclosed herein receive test results from the specialized machinery via one or more connection selected from: a network, an Ethernet, an Internet, a cable, a phone line, a non-transitory computer-readable media, an online cloud, a database, a phone line, an aux connection, a Bluetooth connection, a aux connection, or the like.

In some embodiments, test-retest similarity, test-retest precision, test-retest reliability, and test-retest repeatability are interchangeable or equivalent as disclosed herein. In some embodiments, test-retest similarity, test-retest precision, test-retest reliability, and test-retest repeatability indicate how similar the test-retest results are. In some cases, the test-retest reliability helps determine if the difference in test-retest is of medical significance or not. In some cases, the test-retest reliability helps determine if the difference in test-retest is caused by testing noise, testing errors, and or any other factors unrelated to medical conditions or therapeutic interventions.

In some embodiments, the test-retest precision is a measure of how similar a test-retest(s) pair is. In some embodiments, the test-retest precision is an assessment of the similarity of test-retest results from one subject. In some embodiments, the test-retest precision is an assessment of the similarity of test-retest results from more than one subject, with the test-retest precision assessed for each of the subjects so each subject is the “one subject” once in a test-retest precision measurement. Each test-retest pair may include one test and one retest for the same subject. Each test-retest pair may include one test and one retest for two different subjects.

Each test-retest pair may include one test and more than one retest of the same subjects. Each test-retest pair may include one test result and more than one retest results of the same subjects.

In some embodiments, CoR or the Bland-Altman CoR is calculated as

In some embodiments, ICC is calculated as

In some embodiment, a vision based feature is any feature that may be tested or examined using a vision-related testing device. In some embodiments, non-limiting examples of a feature include but is not limited to one or more selected from: a retinal layer thickness, a retinal blood vessel thickness, a retinal haemorrhage index, a structural descriptor of an image of the eye, a cup-to-disc ratio, a drusen count, an edema count, a frequency doubling technology-related feature, a perimetry subfield threshold, a visual field loss summary statistic, a visual field sensitivity map, an interocular pressure, a saccadic latency, a smooth pursuit gain index, a dark-adaptation rod intercept time, a chromatic contrast threshold, and a temporal frequency threshold. In some embodiments, non-limiting examples of features include one or more selected from: a parameter of acuity, an acuity threshold, an acuity slope, a sensitivity parameter, a sensitivity parameter of CSF, a CSF distribution, a probability distribution of CSF, the peak spatial frequency fmax of CSF, peak sensitivity ymax of CSF, bandwidth β of CSF, and a low-frequency truncation parameter δ of CSF, a sensitivity parameter of quick CSF (qCSF), qCSF, a qCSF distribution, a joint distribution of qCSF over four parameters (a distribution of results in a four-dimensional space, with each result describing a CSF curve and each curve living in a two-dimensional space of spatial frequency and contrast), an area under the Log CSF (AULCSF) integrated over the spatial frequency range from at least 1.5 to 18 cycles per degree (cpd), a CSF acuity, as a non-limiting example, the spatial frequency point where the CSF described by four parameters reaches a contrast threshold of 100%, a sensitivity threshold of the CSF that is described by four parameters for at least one spatial frequency ranging from 1 to 50 cpd, and the spatial point for which the CSF reaches a pre-determined contrast threshold.

In some embodiments, a contrast threshold is the inverse of sensitivity. In some embodiments, the four parameters of CSF or qCSF are a peak spatial frequency fmax of CSF, peak sensitivity γmax of CSF, bandwidth β of CSF, and a low-frequency truncation parameter δ of CSF. In some embodiments, the at least one spatial frequency ranging from 1 to 18 cpd includes at least 1 cpd, 1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and 18 cpd. In some embodiments, AULCSF represents the median, average, or processed AULCSF computed over spatial frequencies ranging from 1.5 cpd to 18 cpd. In some embodiments, the CSF is a curve in a two dimensional space of spatial frequency and contrast.

In further embodiments, a feature includes at least one transformation or an operation of at least one vision feature. In some embodiments, a transformation or an operation includes one or more selected from: a mean, a median, a standard deviation, a multiplication by a pre-selected number or another feature, a square, a cube, a nth order multiplication, a square root, a nth root, a division by a pre-selected number, a random subsampling, a Fourier Transform, a rounding, and a change in quantization level.

In some embodiments, a feature includes a probability or statistical distribution of at least one feature. In some embodiments, a feature includes a joint probability or statistical distribution of at least two features. In some embodiments, a feature includes information derived from a probability distribution, a statistical distribution of one feature, a joint probability distribution of at least two features, a joint statistical distribution of at least two features. In some embodiments, a feature includes a scalar feature.

In some embodiments, the methods and systems disclosed herein includes a feature space. In some embodiments, a feature space includes any combination of the above-mentioned features and/or transformations, where each feature corresponds to one dimension in a multi-dimensional space. In some embodiments, the feature space includes a dimension size no less than one. In some embodiments, at least a number of dimensions in the feature space are orthogonal to each other, the number being no less than 2.

In some embodiments, the feature space undergoes another transformation that scales and rotates the coordinate system of the feature space. As a non-limiting example, individual dimension (feature) is scaled with different factors to make the range of feature values comparable. In some embodiments, the coordinate system is rotated by a whitening step that de-correlates the features. In some embodiments, a principle component analysis (PCA) is applied that de-correlates the features and reduces the dimensionality of the feature space to preserve only the principle components of the feature space. In some embodiments, a test result is represented in a feature space as one multi-dimensional vector, or as a distribution of multi-dimensional vectors.

In some embodiments, in the visual domain, the predominant tool for functional assessment is acuity, which measures the smallest size of a stimulus (typically, a letter) that an observer can recognize at full contrast. In some embodiments, while acuity is well established and can be very useful e.g. for adjusting optical correction, it has at least two issues in domains where precision is paramount, such as in clinical trials. First, the variability of repeated measurements makes the method insensitive to subtle changes in vision, e.g. due to disease progression or treatment. Second, some ophthalmic and neurologic conditions affect acuity only moderately, despite of effects on visual function. In some embodiments, the contrast sensitivity function (CSF) relates an observer's ability to recognize a spatial pattern not only to its size, but also to its contrast. In some embodiments, comparing to acuity, the CSF correlates better with performance in visually guided everyday activities, for non-limiting examples, driving, walking, and the ability to recognize faces.

Traditional tests such as paper charts typically only return one number (e.g. “20/20 vision”, i.e. 100%). In some embodiments, for more complex tests such as the qCSF, which returns joint probability distributions over four parameters (a distribution of test results in a four-dimensional space, with each result describing a CSF curve, each of these curves living in a two-dimensional space of spatial frequency and contrast), an arbitrary number of features is generated based on these distributions to create a feature space of arbitrary dimensionality; for example, the median AULCSF is a one-dimensional feature space, and the combination of median AULCSF and median CSF acuity would give a two-dimensional feature space.

In some embodiments, visual sensitivity is comprehensively described by CSF, but current routine clinical care does not include its assessment because of the time-consuming need to estimate thresholds for a large number of spatial frequencies. In some embodiments, the quick CSF (qCSF) method dramatically reduces testing times by using a Bayesian information maximization rule. In some embodiments, the qCSF method uses a computationally intensive algorithm and optimizes stimulus selection by computing the expected information gain over a very large set of possible stimuli and the probability distribution of possible CSFs, given the history of previous trials. In some embodiments, the CSF is a description of visual function that assigns a contrast sensitivity (the most subtle gray/gray difference that can be discerned) to every possible spatial frequency (size of the stimulus), thus, the CSF space is two-dimensional (contrast and spatial frequency). People have measured the full CSF before, but the combinatorial complexity of the additional dimension made those tests impractical for clinical care—they took 30-60 minutes; the method developed by Lesmes and colleagues, however, reduced testing times to 2-3 minutes (Lesmes et. al. “Bayesian adaptive estimation of the contrast sensitivity function: The quick CSF method,” Journal of Vision 2010 10(3) hence the ‘quick’ CSF).

In some embodiments, the quantization level may be changed by changing the number of digits in a test and a retest score. In other embodiments, the quantization level is not changed by changing the number of digits but may be determined by the number of different test scores that can result from the test. As an example, a test with three scores, ⅓, 1/7, and 1, even with infinite number of digits the test itself provides only three scores thus only very quantized outcomes. In some embodiments, the number of digits is the number of fractional digits. In some embodiments, at least two to at least 5 intermediate steps are included between two adjacent digits after the decimal point. In some cases, as a non-limiting example, a quantization level with 5 fractional digits is less coarse and less quantized than a quantization level with 4 fractional digits given that the fraction digits are for an identical digit after the decimal point. In some embodiments, as a non-limiting example, with 5 fractional steps for the second digit after the decimal point, 1.028 is rounded to 1.02 and 1.071 is rounded to 1.08).

Similarities

The methods and systems disclosed herein, in various embodiments, are applied in order to evaluate similarity, repeatability, reliability, precision, or variability of a test-retest pair. In some cases, the similarity, repeatability, reliability, precision, or variability of a test-retest pair is equivalent or interchangeable as disclosed herein.

In some cases, the similarity of a test-retest pair is assessed by ranking distances between the selected test result and each retest result of the plurality of subjects and obtaining a rank of the distance from the selected test result to the retest result of the same subject. As a nonlimiting example inFIG. 1, a similarity of test-retest pair of subject2(left) is lower than the similarity of test-retest pair of subject3(right), as the distance of test-retest pair of subject2(left) ranks 3 out of 4 while the distance of test-retest pair of subject3(right) ranks 1 out of 4. A rank precision may be obtained based on the similarities. In this case, the rank precision for subject2using MAP is ⅓ and the rank precision for subject3using MAP is 1.

Ranks

In some embodiments, the method described herein includes a rank. In some embodiments, a rank is given to each retest result. Further, each retest result may include one or more retests for each of the plurality of subjects participated in the test-retest precision measurement. In some embodiments, a rank is a real number ranging from 0 to N−1, wherein N is the total number of subjects, the total number of retests, the total number of tests, or the total number of tests and retests. In some embodiments, a rank is a real number ranging from 1 to N, wherein N is the total number of subjects, the total number of retests, the total number of tests, or the total number of tests and retests. In some embodiments, the rank is given to each retest according to a distance of each retest to at least one specified test. As a nonlimiting example, a total of 6 subjects each have one test and two retests results. A rank may be given to each of the 12 retest results based on their distance to the test of the first subject.

Other example of ranks may be seen inFIG. 1. Referring toFIG. 1, in a particular embodiment, four subjects are measured in a test-retest event and test-retest precision is evaluated. In this embodiment, test and retest scores for each of the four subjects are plotted, and the distance of each retest result to the test result of subject2(left panel) and subject3(right panel) is obtained. Correspondingly, in the same embodiment, the rank of distance is optionally calculated. In this embodiment, for subject2, retest score has a rank of 3 in a total number of 4 subjects, so that the average precision of subject2's retest is the inverse of rank of the retest, thus, the precision is ⅓. In the same embodiment, similarly, for subject3, retest score has a rank of 1 in a total number of 4 subjects, so that the average precision of subject3's retest is the inverse of rank of the retest, thus, the precision is 1.

In some embodiments, the rank number is reversibly related to the distance value. As a non-limiting example, the retest result with the shortest distance to at least one specified test result has a rank of 0; the retest result with the longest distance to at least one specified test result has a rank of N−1. In some embodiments, the retest results and/or subjects of the retest results are sorted in a list with the corresponding distance in a monotonically non-descending order: the retest result and/or subject at the top of the sorted list has a rank of 0; the retest result and/or subject at the bottom of the sorted list has a rank of N−1. In some embodiments, the rank is normalized by the total number of subjects. In some embodiments, the rank is weighted given preselected weighting for at least one rank value.

In some embodiments, a rank is any number ranging from 1 to N, wherein N is the total number of subjects, the total number of retest, or the total number of tests. In some embodiments, the rank number is reversibly related to the distance value. As a non-limiting example, the retest result with the shortest distance to at least one specified test result has a rank of 1; the retest result with the longest distance to at least one specified test result has a rank of N. In some embodiments, the retest results and/or subjects of the retest results are sorted in a list with the corresponding distance in a monotonically non-descending order: the retest result and/or subject at the top of the sorted list has a rank of 1; the retest result and/or subject at the bottom of the sorted list has a rank of N.

In some embodiments, the rank is the same for at least two retest results if the distance is identical for at least two retest results and/or subjects. In some embodiments, the identical rank is calculated as an average of adjacent ranks which would be given to the at least two retest results if they were not identical without affecting the ranks of other retest results. In some embodiments, as a non-limiting example, distances of 0.1, 0.1, 0.2, and 0.3 get ranked as 1.5, 1.5, 3, and 4, respectively.

Mean Average Precision (MAP) and Fractional Rank Precision (FRP)

The methods and systems disclosed herein may include one or more suitable test-retest precision measurement methods. In some embodiments, the methods and systems disclosed herein include a MAP or a FRP calculation for the evaluation of test-retest precision, test-retest repeatability, test-retest reliability, or test-retest variation. In some cases, the methods and systems disclosed herein may include a combination of MAP and FRP calculation, the weighting of MAP or FRP calculation may be preselected based on empirical experience, simulation, trails, or any other suitable factors.

In some cases, fractional rank precision may not be limited to non-integer ranking to break ties. In some cases, fractional ranking may also include normalized ranking precision or normalized ranking. In some embodiments, fractional ranking is equivalent to and interchangeable with normalized ranking. In some embodiments, fractional ranking precision is equivalent to and interchangeable with normalized rank precision (NRP).

In some embodiments, with one test and one retest for each subject, rank precision is directly calculated as (1−rank(Ri)/N), wherein the rank goes from 0 to N−1, wherein N is the total number of subjects, and wherein i is the selected subject.

In some embodiments, the FRP is the mean of the rank precision for each subject of all the subjects in a test-retest event.

In some embodiments, with at least one test and at least one retest for each subject, the average precision (AveP) is directly calculated as 1/rank (Ri), wherein rank goes from 1 to N, wherein N is the total number of subjects, and i is the selected subject. The average precision may be interchangeable or equivalent to the ranks precision in various embodiments as disclosed herein. In some embodiment, each subject has more than one test results and more than one retest result, and the AveP is the average of 1/rank(Ri) for each retest of the selected subject, wherein i is the selected subject. In some embodiments, the MAP is the mean of AveP for each subject of all the subjects in a test-retest event.

In some embodiments, the MAP is calculated as

In some embodiments, the FRP is calculated as

FRP=1-1N⁢∑n=1N⁢⁢rankxn,1⁡(xn,2)-1N,
wherein each subject xnhas a test-retest pair, xn,1and xn,2, the total number of subjects is N.

In some embodiments, the methods and systems disclosed herein include one or more recursive steps. In further cases, one or more recursive steps repeats only once for each of the plurality of subjects. In further cases, one or more recursive steps repeats for more than once for each of the plurality of subjects when each subject has more than one retest results, the repetition number being equal to the number of retests for individual subjects.

Distances

In some embodiments, the distance between test-retest results is represented by a number, or a multi-dimensional vector. In some embodiments, the distance may be any distance calculated using any suitable calculation methods for two scalars or vectors representing the test-retest results pair in one or two feature spaces. In some embodiments, a distance is one or more selected from: the Euclidean distance, the Mahalanobis distance, and the Manhattan distance. In some embodiments, the Manhattan distance represents a sum of dimension-wise absolute differences.

In some embodiments, the distance between test results is represented by one or more selected from: the distributions of multi-dimensional vectors, the mean, the median, or other quantiles of the distribution of Euclidean or Manhattan distances between the vectors. In some embodiments, the distance between test-retest results is represented by the mean, median, or any other quantiles of the distribution of Mahalanobis distances between sampled points of the one distribution and the whole other distribution. In some embodiments, the Mahalanobis distance expresses distance in terms of the width of the data ellipsoid in the multi-dimensional feature space.

Referring toFIG. 1, in a particular embodiment, four subjects are measured in a test-retest event and test-retest precision is evaluated using MAP. In this embodiment, test and retest scores for each of the four subjects are plotted, and the distance of each retest result to the test result of subject2(left panel) and subject3(right panel) is obtained. Correspondingly, in the same embodiment, the rank of distance is optionally calculated. In this embodiment, MAP for all four subjects is optionally calculated as the mean of average precision for each of the four subjects. In this case, MAP of all four subjects is 0.708.

Referring toFIG. 2, in a particular embodiment, effect of quantization of test and retest scores on different of test-retest precision (TRP) measures is examined. In this embodiment, a median AULCSF computed over spatial frequencies from 1.5 cpd to 18 cpd is measured using different test-retest precision measures. In this embodiment, the quantization level affects accuracy in test-retest precision measures including CoR, ICC, FRP, and MAP (FIG. 2). In this embodiment, MAP precision score, which reflects measurement precision using MAP, gradually decreases monotonically with increased quantization. Additionally, in the same embodiment, coarse quantization that renders a test uninformative may lead to ‘perfect’ reliability as measured by CoR and ICC. For example, when quantization level is less than 1, ICC may show an inaccurately high precision score. In this embodiment, for intermediate quantization level of about 0.10 to about 1.00, MAP is more sensitive and accurate than ICC.

Referring toFIG. 3, in a particular embodiment, boxplots of the distributions of FRP are plotted over the subjects for various visionary features. The mean of the distribution represents the TRP value for various visionary features. In this embodiment, using FRP measure, test-retest precision may be different when it is based on different vision-based feature. AULCSF may be slightly more accurate with a high average precision while 1.0 cpd may have a lower test-retest precision.

Referring toFIG. 4, in a particular embodiment, effect of quantization on different test-retest precision measures is examined. In this embodiment, a median AULCSF computed over spatial frequencies from 1.5 cpd to 18 cpd is measured using different test-retest precision (TRP) measures. These different measures include CoR, ICC, MAP, and FRP. In this embodiment, CoR is particularly vulnerable to quantization differences and shows non-monotonic behavior with increased level of quantization. In the same embodiment, the ICC also behaves non-monotonically with increased level of quantization. In the same embodiment, the change of FRP evaluation of precision stays monotonic as the level of quantization increases. In this embodiment, the quantization level affects the accuracy in test-retest precision measures including CoR, ICC, MAP, and FRP.

Referring toFIG. 7, in a particular embodiment, the effect of choosing a different prior over the CSF search space on FRP, is determined as a function of trial number. In this case, both the uniform and the population prior lead to very similar FRP results over the first few trials, as only very little information about the true CSF is available. In the same embodiments, for the time range between 10 to 30 trials, the analysis initialized with the population prior shows better convergence, In the same embodiment, for the time range after 30 trials, the difference between FRP diminishes as both approaches converge; however, there is still a small benefit for population after 50 trials (FRP of 0.871 and 0.864, respectively).

Digital Processing Device

In some embodiments, the digital processing device includes a display to send visual information to a user. In some embodiments, the display is a cathode ray tube (CRT). In some embodiments, the display is a liquid crystal display (LCD). In further embodiments, the display is a thin film transistor liquid crystal display (TFT-LCD). In some embodiments, the display is an organic light emitting diode (OLED) display. In various further embodiments, on OLED display is a passive-matrix OLED (PMOLED) or active-matrix OLED (AMOLED) display. In some embodiments, the display is a plasma display. In other embodiments, the display is a video projector. In still further embodiments, the display is a combination of devices such as those disclosed herein.

Referring toFIG. 8, in a particular embodiment, an exemplary digital processing device101is programmed or otherwise configured to measure test-retest reliability using FRP, MAP or other suitable precision evaluation methods. The device101can regulate various aspects of test-retest precision measurement of the present disclosure, such as, for example, formulating test-retest reliability as an information retrieval problem, and ranking retest measurements by their distance to a subject's test measurement. As another example, it may assess a similarity between a test result and a retest result. In this embodiment, the digital processing device101includes a central processing unit (CPU, also “processor” and “computer processor” herein)105, which can be a single core or multi core processor, or a plurality of processors for parallel processing. The digital processing device101also includes memory or memory location110(e.g., random-access memory, read-only memory, flash memory), electronic storage unit115(e.g., hard disk), communication interface120(e.g., network adapter) for communicating with one or more other systems, and peripheral devices125, such as cache, other memory, data storage and/or electronic display adapters. The memory110, storage unit115, interface120and peripheral devices125are in communication with the CPU105through a communication bus (solid lines), such as a motherboard. The storage unit115can be a data storage unit (or data repository) for storing data. The digital processing device101can be operatively coupled to a computer network (“network”)130with the aid of the communication interface120. The network130can be the Internet, an internet and/or extranet, or an intranet and/or extranet that is in communication with the Internet. The network130in some cases is a telecommunication and/or data network. The network130can include one or more computer servers, which can enable distributed computing, such as cloud computing. The network130, in some cases with the aid of the device101, can implement a peer-to-peer network, which may enable devices coupled to the device101to behave as a client or a server. The digital processing device101can be operatively connected to one or more specialized medical device (not shown) via the network130. Such connection may enable data collection from the medical device; the data may include one or more test results, retest results, and other related test and subject information. The specialized medical device is configured to measure visionary features(s) of one or more subjects.

Continuing to refer toFIG. 8, the CPU105can execute a sequence of machine-readable instructions, which can be embodied in a program or software. The instructions may be stored in a memory location, such as the memory110. The instructions can be directed to the CPU105, which can subsequently program or otherwise configure the CPU105to implement methods of the present disclosure. Examples of operations performed by the CPU105can include fetch, decode, execute, and write back. The CPU105can be part of a circuit, such as an integrated circuit. One or more other components of the device101can be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

Continuing to refer toFIG. 8, the storage unit115can store files, such as drivers, libraries and saved programs. The storage unit115can store user data, e.g., user preferences and user programs. The digital processing device101in some cases can include one or more additional data storage units that are external, such as located on a remote server that is in communication through an intranet or the Internet.

Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the digital processing device101, such as, for example, on the memory110or electronic storage unit115. The machine executable or machine readable code can be provided in the form of software. During use, the code can be executed by the processor105. In some cases, the code can be retrieved from the storage unit115and stored on the memory110for ready access by the processor105. In some situations, the electronic storage unit115can be precluded, and machine-executable instructions are stored on memory110.

Non-Transitory Computer Readable Storage Medium

Computer Program

Web Application

Mobile Application

Those of skill in the art will recognize that several commercial forums are available for distribution of mobile applications including, by way of non-limiting examples, Apple® App Store, Google® Play, Chrome Web Store, BlackBerry® App World, App Store for Palm devices, App Catalog for webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia® devices, Samsung® Apps, and Nintendo® DSi Shop.

Standalone Application

Web Browser Plug-In

Software Modules

Databases

In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more databases, or use of the same. In view of the disclosure provided herein, those of skill in the art will recognize that many databases are suitable for storage and retrieval of information including one or more selected from: one or more of subjects, one or more test types, one or more retest types, one or more test results associated with each subject, one or more retests associated with each subject, and one or more features for the test/retest results. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity-relationship model databases, associative databases, and XML databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, and Sybase. In some embodiments, a database is internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloud computing-based. In other embodiments, a database is based on one or more local computer storage devices.

INCORPORATION BY REFERENCE

Certain Definitions

EXAMPLES

The following illustrative examples are representative of embodiments of the software applications, systems, and methods described herein and are not meant to be limiting in any way.

Comfortably seated, subjects holding an iPad 4 that is set to a mean screen luminance of 185 cd/m2 at a viewing distance of 60 cm. In each vision test-retest trial, a randomly chosen bandpass-filtered Sloan letter is presented for 500 ms, and the subject then had to register their response by touch-selecting a letter from an array on the screen (10-Alternatives Forced Choice). The stimulus space is log-spaced and comprised 24 spatial scales (0.64 to 41 cycles per degree (cpd)) and 48 contrast levels (0.2 to 100%). In order to avoid uninformative regions of the stimulus space, the next stimulus was then randomly chosen from the top 10 percent of the distribution of expected information gain. All subjects run the test multiple times. Controls are tested monocularly in both eyes and binocularly; to estimate reliability of the procedure, one of these three conditions (randomly chosen) is repeated. Myopes are tested both with and without their optical correction, with one repeat, for a total of 7 conditions. We report only on the repeated data.

Two traditional methods, ICC and CoR are compared with the FRP method described herein for their measurements of the test-retest precision of different features for vision sensitivity testing. Such vision sensitivity testing features include AULCSF, which is the area under the log CSF integrated between spatial frequencies from 1.5 to 18 cpd, the CSF acuity, and the sensitivity threshold of CSF at six different spatial frequencies including 1 cpd, 1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and 18 cpd. Results of comparison are shown in Table 1. As shown inFIG. 3, boxplots of the distributions of FRP are plotted over a plurality of subjects for various visionary features that may be included in a feature space. These various features are non-limiting examples of vision-based features that may be used to measure qualitative or quantitative aspects of human vision. The mean of the distribution represents the TRP value for each feature. FRP method disclosed herein indicates that AULCSF most precisely identifies test-retest pairs even though it summarizes over a broad range of spatial frequencies. Additionally, the sensitivity at 6 cpd has a much higher CoR, 0.293, than the CSF acuity feature, 0.193, but more precisely identifies test-retest pairs according to the FRP method disclosed herein, 0.848 vs. 0.844. The ICC is vulnerable to outliers: adding a single subject with very poor vision (for non-limiting example, test-retest sensitivity at 1.0 cpd of 0.1 and 0.05 respectively) substantially changes ICC from 0.868 to 0.817; while FRP is much less affected by outliers, and the change is less than 0.005.

Bland-Altman CoR and ICC for two selected features are graphically shown inFIG. 5.

Referring toFIG. 5, in a particular embodiment, test-retest precision is evaluated for test-retest scored using features including AULCSF and sensitivity of CSF at 1.0 cpd. In this embodiment, test-retest scores are compared using traditional Bland-Altman plot (top panels) and correlation plot (bottom panels). In this embodiment, dashed lines represent bias and 95% confidence intervals for test-retest differences. In this embodiment, the Bland-Altman plots do not show obvious dependencies of test-retest differences on the magnitude of the mean test score. The standard deviation of differences for 1.0 cpd is slightly smaller than for AULCSF. In the same embodiment, scatter correlation plots in the bottom panels also show strong agreement between test and retest scores.

Referring to Table 1, in a particular embodiment, fractional rank precision is calculated for different vision sensitivity features including AULCSF, which is the area under the log CSF between spatial frequencies from 1.5 to 18 cpd, the CSF acuity, and the sensitivity threshold of CSF at six different spatial frequencies including 1 cpd, 1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and 18 cpd. In this embodiment, the AULCSF shows the largest area under the precision-recall curve, thus, the greatest fractional rank precision among all the examined features. In this embodiment, the AULCSF has the smallest test-retest variability and greatest test-retest precision.

Fractional rank precision values for different features of qCSF are examined in test-retest trials with multiple subjects. As shown inFIG. 6, with dashed lines indicating chance level (0.5) and FRP for subjective prior assessment (0.767). Among scalar features, the AULCSF yields higher FRP levels than CSF acuity (after 50 trials, FRP of 0.871 and 0.844, respectively) as it summarizes over a large range of spatial frequencies. Multi-dimensional features, however, identify test-retest pairs with greater precision. The combination of high-frequency cutoff (CSF acuity) and the sensitivity near the presumed peak of the CSF (1.5 cpd) already yields a FRP of 0.885; further adding sensitivity at a mid-range spatial frequency (6 cpd) improves FRP to 0.899, and the addition of the AULCSF results in the highest FRP of 0.901. Notably, FRP seems not to have plateaued after 50 trials for the multi-dimensional features indicating gain in precision is obtainable by running the qCSF method for more trials.

Comfortably seated, subjects holding an iPad 4 that is set to a mean screen luminance of 185 cd/m2 at a viewing distance of 60 cm. In each vision test-retest trial, a randomly chosen bandpass-filtered Sloan letter is presented for 500 ms, and the subject then had to register their response by touch-selecting a letter from an array on the screen (10-Alternatives Forced Choice). The stimulus space is log-spaced and comprised 24 spatial scales (0.64 to 41 cycles per degree (cpd)) and 48 contrast levels (0.2 to 100%). In order to avoid uninformative regions of the stimulus space, the next stimulus was then randomly chosen from the top 10 percent of the distribution of expected information. All subjects run the test multiple times. Controls are tested monocularly in both eyes and binocularly; to estimate reliability of the procedure, one of these three conditions (randomly chosen) is repeated. Myopes are tested both with and without their optical correction, with one repeat, for a total of 7 conditions. We report only on the repeated data.

Two traditional methods, ICC and CoR are compared with the MAP method described herein for their measurements of the test-retest precision of different features for vision sensitivity testing. Such vision sensitivity testing features include AULCSF, which is the area under the log CSF between spatial frequencies from 1.5 to 18 cpd, the CSF acuity, and the sensitivity threshold of CSF at six different spatial frequencies including 1 cpd, 1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and 18 cpd. Results of comparison are shown in Table 2. Among all the tested features, AULCSF yields highest MAP. In other words, AULCSF is the best feature to identify a pair of test-retest measurements, even though it summarizes over a broad range of spatial frequencies. Sensitivities for low spatial frequencies including 1 cpd and 1.5 cpd vary less across subjects, therefore are less discriminative across subjects.