A video-based method of measuring a patient's vital sign, comprising: receiving, by a processor, a video signal from a video camera, the video signal having a field of view that includes exposed skin of a patient; extracting, by the processor, from the video signal a time-varying color signal for each of a plurality of regions, each region including exposed skin of the patient; identifying, by the processor, a frequency content of each color signal; selecting, by the processor, a plurality of candidate regions including exposed skin of said patient and each having a shared frequency content comprising a modulation at a shared frequency; selecting, by the processor, from the plurality of candidate regions having the shared frequency content comprising the modulation at the shared frequency two or more non-adjacent regions; combining, by the processor, the color signals of the selected non-adjacent regions to produce a combined color signal; calculating, by the processor, a statistic of the combined color signal, wherein the statistic comprises an amplitude, a variability, a skew, or a signal to noise ratio; measuring, by the processor, a vital sign from the combined color signal; determining, by the processor, a weight for the vital sign, the weight being based on the statistic, and adding the vital sign to a running average based on the weight to obtain a final value for the vital sign; and outputting, by the processor, the final value for the vital sign for further processing or display, wherein the method further comprises updating the selected regions by adding or removing pixels or regions from the selected regions based on the frequency content of each region, thereby to remove pixels or regions that do not satisfy a criteria for vital sign measurement, and to add new pixels or regions that do satisfy the criteria. receiving, by a processor, a video signal from a video camera, the video signal having a field of view that includes exposed skin of a patient; extracting, by the processor, from the video signal a time-varying color signal for each of a plurality of regions, each region including exposed skin of the patient; identifying, by the processor, a frequency content of each color signal; selecting, by the processor, a plurality of candidate regions including exposed skin of said patient and each having a shared frequency content comprising a modulation at a shared frequency; selecting, by the processor, from the plurality of candidate regions having the shared frequency content comprising the modulation at the shared frequency two or more non-adjacent regions; combining, by the processor, the color signals of the selected non-adjacent regions to produce a combined color signal; calculating, by the processor, a statistic of the combined color signal, wherein the statistic comprises an amplitude, a variability, a skew, or a signal to noise ratio; measuring, by the processor, a vital sign from the combined color signal; determining, by the processor, a weight for the vital sign, the weight being based on the statistic, and adding the vital sign to a running average based on the weight to obtain a final value for the vital sign; and outputting, by the processor, the final value for the vital sign for further processing or display, wherein the method further comprises updating the selected regions by adding or removing pixels or regions from the selected regions based on the frequency content of each region, thereby to remove pixels or regions that do not satisfy a criteria for vital sign measurement, and to add new pixels or regions that do satisfy the criteria.