Apparatus for decoding an encoded audio signal to obtain modified output signals, comprising: an input interface for receiving the encoded audio signal and for extracting, from the encoded audio signal, a transmitted downmix signal and parametric data relating to audio objects included in the transmitted downmix signal, the transmitted downmix signal being different from an encoder downmix signal, to which the parametric data is related, wherein the encoder downmix signal is generated by an encoder by downmixing the audio objects using downmixing information; a downmix modifier for modifying the transmitted downmix signal using a downmix modification function, wherein the downmix modification function is such that a modified downmix signal is identical to the encoder downmix signal or is more similar to the encoder downmix signal compared to the transmitted downmix signal, wherein the downmix modification function comprises applying downmix modification gain factors or interpolated or smoothed downmix modification gain factors to different time frames or frequency bands of the transmitted downmix signal; and an object renderer for rendering the audio objects using the modified downmix signal and the parametric data to obtain output signals; characterized by an output signal modifier for modifying the output signals using an output signal modification function, wherein the output signal modification function is such that a manipulation operation applied to the encoder downmix signal to obtain the transmitted downmix signal is at least partly applied to the output signals to obtain the modified output signals, wherein the output signal modification function comprises applying output signal modification gain factors or interpolated or smoothed output signal modification gain factors to different time frames or frequency bands of the output signals, wherein the input interface is configured to additionally receive information on the downmix modification gain factors, and wherein the output signal modifier is configured to derive the output signal modification gain factors from inverse values of the downmix modification gain factors, or wherein the input interface is configured to additionally receive information on the output signal modification gain factors, and wherein the downmix modifier is configured to derive the downmix modification gain factors from inverse values of the output signal modification gain factors. an input interface for receiving the encoded audio signal and for extracting, from the encoded audio signal, a transmitted downmix signal and parametric data relating to audio objects included in the transmitted downmix signal, the transmitted downmix signal being different from an encoder downmix signal, to which the parametric data is related, wherein the encoder downmix signal is generated by an encoder by downmixing the audio objects using downmixing information; a downmix modifier for modifying the transmitted downmix signal using a downmix modification function, wherein the downmix modification function is such that a modified downmix signal is identical to the encoder downmix signal or is more similar to the encoder downmix signal compared to the transmitted downmix signal, wherein the downmix modification function comprises applying downmix modification gain factors or interpolated or smoothed downmix modification gain factors to different time frames or frequency bands of the transmitted downmix signal; and an object renderer for rendering the audio objects using the modified downmix signal and the parametric data to obtain output signals; characterized by an output signal modifier for modifying the output signals using an output signal modification function, wherein the output signal modification function is such that a manipulation operation applied to the encoder downmix signal to obtain the transmitted downmix signal is at least partly applied to the output signals to obtain the modified output signals, wherein the output signal modification function comprises applying output signal modification gain factors or interpolated or smoothed output signal modification gain factors to different time frames or frequency bands of the output signals, wherein the input interface is configured to additionally receive information on the downmix modification gain factors, and wherein the output signal modifier is configured to derive the output signal modification gain factors from inverse values of the downmix modification gain factors, or wherein the input interface is configured to additionally receive information on the output signal modification gain factors, and wherein the downmix modifier is configured to derive the downmix modification gain factors from inverse values of the output signal modification gain factors.