A conference controller (111, 175) configured to place L upstream audio signals (123, 173) associated with respective L conference participants within a 2D or 3D conference scene to be rendered to a listener; L being an integer, L>1; wherein the conference controller (111, 175) is configured to - set up a X-point conference scene with X different spatial talker locations within the conference scene, X being an integer, X>0; - assign the L upstream audio signals (123, 173) to the X talker locations; - determine a maximum number N of downstream audio signals (124, 174) to be transmitted to a listener terminal (120, 170) of the listener; N being an integer; wherein N<X; - determine that the number L of upstream audio signals (123, 173) assigned to the different talker locations within the conference scene is greater than N; - determine, from the L assigned upstream audio signals (123, 173) based on the L upstream audio signals (123, 173), N downstream audio signals (124, 174) to be transmitted to the listener terminal (120, 170) of the listener; wherein determining N downstream audio signals (124, 174) comprises - determining a degree of activity of the L upstream audio signals (123, 173); and - initiating a mixing of at least two upstream audio signals (123, 173) having the at least two lowest degrees of activity, thereby yielding a first of the N downstream audio signals (124, 174); - determine N updated talker locations for the N downstream audio signals (124, 174), respectively, based on the talker locations which the L upstream audio signals (123, 173) are assigned to; - assign the first downstream audio signal (124, 174) to a first of the N updated talker locations; and - generate metadata identifying the updated talker locations and enabling an audio processing unit (121, 171) of the listener terminal (120, 170) of the listener to generate a spatialized audio signal based on the N downstream audio signals (124, 174); wherein when rendering the spatialized audio signal to the listener, the listener perceives the N downstream audio signals (124, 174) as coming from the N updated talker locations, respectively, characterized in that the talker locations are positioned in an asymmetric manner with regards to a midline in front of a head of the listener. - set up a X-point conference scene with X different spatial talker locations within the conference scene, X being an integer, X>0; - assign the L upstream audio signals (123, 173) to the X talker locations; - determine a maximum number N of downstream audio signals (124, 174) to be transmitted to a listener terminal (120, 170) of the listener; N being an integer; wherein N<X; - determine that the number L of upstream audio signals (123, 173) assigned to the different talker locations within the conference scene is greater than N; - determine, from the L assigned upstream audio signals (123, 173) based on the L upstream audio signals (123, 173), N downstream audio signals (124, 174) to be transmitted to the listener terminal (120, 170) of the listener; wherein determining N downstream audio signals (124, 174) comprises - determining a degree of activity of the L upstream audio signals (123, 173); and - initiating a mixing of at least two upstream audio signals (123, 173) having the at least two lowest degrees of activity, thereby yielding a first of the N downstream audio signals (124, 174); - determining a degree of activity of the L upstream audio signals (123, 173); and - initiating a mixing of at least two upstream audio signals (123, 173) having the at least two lowest degrees of activity, thereby yielding a first of the N downstream audio signals (124, 174); - determine N updated talker locations for the N downstream audio signals (124, 174), respectively, based on the talker locations which the L upstream audio signals (123, 173) are assigned to; - assign the first downstream audio signal (124, 174) to a first of the N updated talker locations; and - generate metadata identifying the updated talker locations and enabling an audio processing unit (121, 171) of the listener terminal (120, 170) of the listener to generate a spatialized audio signal based on the N downstream audio signals (124, 174); wherein when rendering the spatialized audio signal to the listener, the listener perceives the N downstream audio signals (124, 174) as coming from the N updated talker locations, respectively, characterized in that the talker locations are positioned in an asymmetric manner with regards to a midline in front of a head of the listener.