A video encoder comprising: a memory unit configured to store one or more direct reference layer pictures of one or more current pictures in a sequence, wherein the one or more current pictures are associated with a current layer, the current layer being associated with the one or more direct reference layers, and wherein a direct reference layer is a layer that may be used for inter-layer prediction of another layer, such as the current layer; and a processor in communication with the memory unit and configured to: determine that a direct reference layer picture of the one or more direct reference layer pictures is not restricted for use, such that the direct reference layer picture can be used, in inter-layer prediction, when the direct reference layer picture has a temporal identification value less than a predetermined threshold based on a maximum temporal identification value associated with the current layer; determine whether all of the one or more direct reference layer pictures of a current picture that are not restricted for use in inter-layer prediction are included in an inter-layer reference picture set associated with the current picture; and set a flag associated with the current picture to a first value based on a determination that all of the direct reference layer pictures of the current picture that are not restricted for use in inter-layer prediction are included in the inter-layer reference picture set associated with the current picture; or set the flag to a second value based on a determination that not all of the direct reference layer pictures of the current picture that are not restricted for use in inter-layer prediction are included in the inter-layer reference picture set associated with the current picture. a memory unit configured to store one or more direct reference layer pictures of one or more current pictures in a sequence, wherein the one or more current pictures are associated with a current layer, the current layer being associated with the one or more direct reference layers, and wherein a direct reference layer is a layer that may be used for inter-layer prediction of another layer, such as the current layer; and a processor in communication with the memory unit and configured to: determine that a direct reference layer picture of the one or more direct reference layer pictures is not restricted for use, such that the direct reference layer picture can be used, in inter-layer prediction, when the direct reference layer picture has a temporal identification value less than a predetermined threshold based on a maximum temporal identification value associated with the current layer; determine whether all of the one or more direct reference layer pictures of a current picture that are not restricted for use in inter-layer prediction are included in an inter-layer reference picture set associated with the current picture; and set a flag associated with the current picture to a first value based on a determination that all of the direct reference layer pictures of the current picture that are not restricted for use in inter-layer prediction are included in the inter-layer reference picture set associated with the current picture; or determine that a direct reference layer picture of the one or more direct reference layer pictures is not restricted for use, such that the direct reference layer picture can be used, in inter-layer prediction, when the direct reference layer picture has a temporal identification value less than a predetermined threshold based on a maximum temporal identification value associated with the current layer; determine whether all of the one or more direct reference layer pictures of a current picture that are not restricted for use in inter-layer prediction are included in an inter-layer reference picture set associated with the current picture; and set a flag associated with the current picture to a first value based on a determination that all of the direct reference layer pictures of the current picture that are not restricted for use in inter-layer prediction are included in the inter-layer reference picture set associated with the current picture; or set the flag to a second value based on a determination that not all of the direct reference layer pictures of the current picture that are not restricted for use in inter-layer prediction are included in the inter-layer reference picture set associated with the current picture.