A method for encoding video data or decoding video data, the method comprising: encoding, with a video encoder, or decoding, with a video decoder, syntax elements of a parameter set, which is one of a sequence parameter set or a picture parameter set, that identify candidate long-term reference pictures, wherein one or more of the candidate long-term reference pictures belong in a reference picture set of a current picture, and wherein the reference picture set identifies all reference pictures including: reference pictures that can potentially be used for inter-predicting the current picture and other reference pictures that cannot be used for inter-predicting the current pictures but that can potentially be used for inter-predicting one or more pictures following the current picture in decoding order; encoding, with the video encoder, or decoding, with the video decoder, syntax elements that indicate which candidate long-term reference pictures, identified in the parameter set, belong in the reference picture set of the current picture; constructing, with the video encoder or the video decoder, a plurality of reference picture subsets that form the reference picture set, and identifying one or more reference pictures in at least one of the constructed plurality of reference picture subsets based on which candidate long-term reference pictures belong in the reference picture set of the current picture, and wherein one of the constructed plurality of reference picture subsets identifies zero or more long-term reference pictures that can potentially be used for inter-predicting the current picture, and another of the constructed plurality of reference picture subsets identifies zero or more long-term reference pictures that cannot be used for inter-predicting the current pictures but that can potentially be used for inter-predicting the one or more pictures following the current picture in decoding order; constructing one or both of a first initial reference picture list and a second initial reference picture list by listing, in the respective one or in both of the first initial reference picture list and the second initial reference picture list, at most all of the reference pictures of those of the constructed plurality of reference picture subsets which identify the reference pictures that can potentially be used for inter-predicting the current picture; modifying the respective one or both of the constructed first initial reference picture list and second initial reference picture list to obtain a first final reference picture list and/or second final reference picture list; and inter-predicting, with the video encoder or the video decoder, the current picture based on one or more reference pictures identified in the first final reference picture list and/or the second final reference picture list. encoding, with a video encoder, or decoding, with a video decoder, syntax elements of a parameter set, which is one of a sequence parameter set or a picture parameter set, that identify candidate long-term reference pictures, wherein one or more of the candidate long-term reference pictures belong in a reference picture set of a current picture, and wherein the reference picture set identifies all reference pictures including: reference pictures that can potentially be used for inter-predicting the current picture and other reference pictures that cannot be used for inter-predicting the current pictures but that can potentially be used for inter-predicting one or more pictures following the current picture in decoding order; encoding, with the video encoder, or decoding, with the video decoder, syntax elements that indicate which candidate long-term reference pictures, identified in the parameter set, belong in the reference picture set of the current picture; constructing, with the video encoder or the video decoder, a plurality of reference picture subsets that form the reference picture set, and identifying one or more reference pictures in at least one of the constructed plurality of reference picture subsets based on which candidate long-term reference pictures belong in the reference picture set of the current picture, and wherein one of the constructed plurality of reference picture subsets identifies zero or more long-term reference pictures that can potentially be used for inter-predicting the current picture, and another of the constructed plurality of reference picture subsets identifies zero or more long-term reference pictures that cannot be used for inter-predicting the current pictures but that can potentially be used for inter-predicting the one or more pictures following the current picture in decoding order; constructing one or both of a first initial reference picture list and a second initial reference picture list by listing, in the respective one or in both of the first initial reference picture list and the second initial reference picture list, at most all of the reference pictures of those of the constructed plurality of reference picture subsets which identify the reference pictures that can potentially be used for inter-predicting the current picture; modifying the respective one or both of the constructed first initial reference picture list and second initial reference picture list to obtain a first final reference picture list and/or second final reference picture list; and inter-predicting, with the video encoder or the video decoder, the current picture based on one or more reference pictures identified in the first final reference picture list and/or the second final reference picture list.