A method for processing an audio signal, comprising: receiving a downmix signal including a number of channels and receiving side information including downmix gain information; extracting extension type identifier indicating whether extension area includes a residual signal from the side information; when the extension type identifier indicates that the extension area includes the residual signal, extracting control restriction information for a residual using mode from the side information, wherein the residual using mode is a mode of generating an output signal including a number of channels using the residual signal; receiving control information for controlling gain or panning of at least one object signal; estimating modified control information based on a condition for the residual using mode using the control information, the control restriction information and a downmix matrix, wherein the condition for the residual using mode is that all of the regular object signals are controllable as one instead of being controlled individually and the number of channels of the downmix signal is the same as the number of channels of the output signal, and wherein the downmix matrix is generated using the downmix gain information including a gain indicating an extent of each object included in the downmix signal; obtaining at least one of an enhanced object signal and one or more regular object signals from the downmix signal using the residual signal; and, generating an output signal using the modified control information and at least one of enhanced object signal and one or more regular object signal, wherein the control restriction information for residual using mode relates to a parameter indicating limiting degree of the control information in case of the residual using mode, wherein the enhanced object signal corresponds to an accompanied music, and wherein the regular object signal corresponds to a vocal. receiving a downmix signal including a number of channels and receiving side information including downmix gain information; extracting extension type identifier indicating whether extension area includes a residual signal from the side information; when the extension type identifier indicates that the extension area includes the residual signal, extracting control restriction information for a residual using mode from the side information, wherein the residual using mode is a mode of generating an output signal including a number of channels using the residual signal; receiving control information for controlling gain or panning of at least one object signal; estimating modified control information based on a condition for the residual using mode using the control information, the control restriction information and a downmix matrix, wherein the condition for the residual using mode is that all of the regular object signals are controllable as one instead of being controlled individually and the number of channels of the downmix signal is the same as the number of channels of the output signal, and wherein the downmix matrix is generated using the downmix gain information including a gain indicating an extent of each object included in the downmix signal; obtaining at least one of an enhanced object signal and one or more regular object signals from the downmix signal using the residual signal; and, generating an output signal using the modified control information and at least one of enhanced object signal and one or more regular object signal, wherein the control restriction information for residual using mode relates to a parameter indicating limiting degree of the control information in case of the residual using mode, wherein the enhanced object signal corresponds to an accompanied music, and wherein the regular object signal corresponds to a vocal.