A method of encoding video data in a sequence of frames including a first frame of reference video, the method comprising: interpolating first pixel values for the first frame of reference video to generate first prediction data, wherein interpolating the first pixel values is based on a first set of interpolation filters; inserting a syntax element into a sequence header to signal to a video decoder whether the frames of the sequence were coded using a pre-defined switched interpolation filter set or a new set of interpolation filters and, if the syntax element signals that a new set of interpolation filters is used, then generating subsequent data into the sequence header to transmit the new set of filters, and if the syntax element indicates that the frames of the sequence were encoded using the pre-defined switched interpolation filter set, then subsequent data describing filter coefficients are not included in the sequence header; generating a first syntax element associated with the first frame of reference video to indicate whether the first set of interpolation filters are defined in a syntax element in a sequence header of the sequence of frames, whether the first set of interpolation filters is the same set of interpolation filters as in a previous frame, or if the first set of interpolation filters is a frame-specific set of interpolation filters; including, if the first frame was encoded using a frame-specific interpolation filter, the coefficients of the frame-specific interpolation filters in a frame header; encoding the video data based on the first prediction data; generating an encoded bit stream that includes the encoded video data, the sequence header and the first syntax element in the frame header. interpolating first pixel values for the first frame of reference video to generate first prediction data, wherein interpolating the first pixel values is based on a first set of interpolation filters; inserting a syntax element into a sequence header to signal to a video decoder whether the frames of the sequence were coded using a pre-defined switched interpolation filter set or a new set of interpolation filters and, if the syntax element signals that a new set of interpolation filters is used, then generating subsequent data into the sequence header to transmit the new set of filters, and if the syntax element indicates that the frames of the sequence were encoded using the pre-defined switched interpolation filter set, then subsequent data describing filter coefficients are not included in the sequence header; generating a first syntax element associated with the first frame of reference video to indicate whether the first set of interpolation filters are defined in a syntax element in a sequence header of the sequence of frames, whether the first set of interpolation filters is the same set of interpolation filters as in a previous frame, or if the first set of interpolation filters is a frame-specific set of interpolation filters; including, if the first frame was encoded using a frame-specific interpolation filter, the coefficients of the frame-specific interpolation filters in a frame header; encoding the video data based on the first prediction data; generating an encoded bit stream that includes the encoded video data, the sequence header and the first syntax element in the frame header.