A method of processing a compressed representation of a spatial audio signal for generating a reconstructed representation of the spatial audio signal, wherein the compressed representation comprises a channel-based audio signal with a predefined number of channels and metadata, the metadata comprising direction information and energy information, with the direction information comprising indications of directions of arrival of one or more audio elements in an audio scene and the energy information comprising, for at least one frequency subband, respective indications of signal power associated with the directions of arrival, the method comprising: generating audio signals of the one or more audio elements based on the channel-based audio signal, the direction information, and the energy information; and generating a residual audio signal from which the one or more audio elements are substantially absent, based on the channel-based audio signal, the direction information, and the energy information, wherein generating audio signals of the one or more audio elements comprises: determining coefficients of an inverse mixing matrix M for mapping the channel-based audio signal to an intermediate representation comprising the residual audio signal and the audio signals of the one or more audio elements, based on the direction information and the energy information, wherein determining the coefficients of the inverse mixing matrix M comprises: determining, for each of the one or more audio elements, a panning vector Pan down ( dir) for panning the audio element to the channels of the channel-based audio signal, based on the direction of arrival dir of the audio element; determining a mixing matrix E that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal, based on the determined panning vectors; determining a covariance matrix S for the intermediate representation based on the energy information; and determining the coefficients of the inverse mixing matrix M based on the mixing matrix F and the covariance matrix S. generating audio signals of the one or more audio elements based on the channel-based audio signal, the direction information, and the energy information; and generating a residual audio signal from which the one or more audio elements are substantially absent, based on the channel-based audio signal, the direction information, and the energy information, wherein generating audio signals of the one or more audio elements comprises: determining coefficients of an inverse mixing matrix M for mapping the channel-based audio signal to an intermediate representation comprising the residual audio signal and the audio signals of the one or more audio elements, based on the direction information and the energy information, wherein determining the coefficients of the inverse mixing matrix M comprises: determining, for each of the one or more audio elements, a panning vector Pan down ( dir) for panning the audio element to the channels of the channel-based audio signal, based on the direction of arrival dir of the audio element; determining a mixing matrix E that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal, based on the determined panning vectors; determining a covariance matrix S for the intermediate representation based on the energy information; and determining the coefficients of the inverse mixing matrix M based on the mixing matrix F and the covariance matrix S. determining coefficients of an inverse mixing matrix M for mapping the channel-based audio signal to an intermediate representation comprising the residual audio signal and the audio signals of the one or more audio elements, based on the direction information and the energy information, wherein determining the coefficients of the inverse mixing matrix M comprises: determining, for each of the one or more audio elements, a panning vector Pan down ( dir) for panning the audio element to the channels of the channel-based audio signal, based on the direction of arrival dir of the audio element; determining a mixing matrix E that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal, based on the determined panning vectors; determining a covariance matrix S for the intermediate representation based on the energy information; and determining the coefficients of the inverse mixing matrix M based on the mixing matrix F and the covariance matrix S. determining, for each of the one or more audio elements, a panning vector Pan down ( dir) for panning the audio element to the channels of the channel-based audio signal, based on the direction of arrival dir of the audio element; determining a mixing matrix E that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal, based on the determined panning vectors; determining a covariance matrix S for the intermediate representation based on the energy information; and determining the coefficients of the inverse mixing matrix M based on the mixing matrix F and the covariance matrix S.