A method for controlling hearing devices, each hearing device being worn by a user (29a, 29b, 29c, 29d) on the head of the user, the users of the hearing devices having a conversation and the hearing devices being communicatively interconnected with each other for data exchange to form a hearing device system, wherein the method comprises: acquiring a sound signal of each user (29a, 29b, 29c, 29d) with a microphone of the hearing device of the user (29a, 29b, 29c, 29d); acquiring spatial information of each user (29a, 29b, 29c, 29d), the spatial information comprising information about a spatial orientation and/or distance of the user (29a, 29b, 29c, 29d) with respect to other users (29a, 29b, 29c, 29d) and wherein the spatial information for a user (29a, 29b, 29c, 29d) comprises information acquired with a sensor of the hearing device of the user (29a, 29b, 29c, 29d) or the spatial information is determined based on a radio signal sent by another hearing device; determining conversation patterns (30a, 30b, 30c) from the spatial information acquired for all users (29a, 29b, 29c, 29d), wherein each conversation pattern (30a, 30b, 30c) comprises at least a probability that a user (29a, 29b, 29c, 29d) is talking to another user (29a, 29b, 29c, 29d), wherein the probability is dependent on how strong the orientation of the head of the user is aligned with a line between the user and the other user; determining conversation groups (32a, 32b) from the conversation patterns (30a, 30b, 30c), wherein the users (29a, 29b, 29c, 29d) are grouped into the conversation groups (32a, 32b) by clustering conversation patterns (30a, 30b, 30c) based on their probabilities; outputting the sound signals from the users (29a, 29b, 29c, 29d) with a hearing device, such that sound signals from users (29a, 29b, 29c, 29d) of a conversation group (32a, 32b), the user (29a, 29b, 29c, 29d) of the hearing device is a member of, are output to the user (29a, 29b, 29c, 29d) and that sound signals from users (29a, 29b, 29c, 29d) of conversation groups (32a, 32b), the user (29a, 29b, 29c, 29d) is not a member of, are muted. acquiring a sound signal of each user (29a, 29b, 29c, 29d) with a microphone of the hearing device of the user (29a, 29b, 29c, 29d); acquiring spatial information of each user (29a, 29b, 29c, 29d), the spatial information comprising information about a spatial orientation and/or distance of the user (29a, 29b, 29c, 29d) with respect to other users (29a, 29b, 29c, 29d) and wherein the spatial information for a user (29a, 29b, 29c, 29d) comprises information acquired with a sensor of the hearing device of the user (29a, 29b, 29c, 29d) or the spatial information is determined based on a radio signal sent by another hearing device; determining conversation patterns (30a, 30b, 30c) from the spatial information acquired for all users (29a, 29b, 29c, 29d), wherein each conversation pattern (30a, 30b, 30c) comprises at least a probability that a user (29a, 29b, 29c, 29d) is talking to another user (29a, 29b, 29c, 29d), wherein the probability is dependent on how strong the orientation of the head of the user is aligned with a line between the user and the other user; determining conversation groups (32a, 32b) from the conversation patterns (30a, 30b, 30c), wherein the users (29a, 29b, 29c, 29d) are grouped into the conversation groups (32a, 32b) by clustering conversation patterns (30a, 30b, 30c) based on their probabilities; outputting the sound signals from the users (29a, 29b, 29c, 29d) with a hearing device, such that sound signals from users (29a, 29b, 29c, 29d) of a conversation group (32a, 32b), the user (29a, 29b, 29c, 29d) of the hearing device is a member of, are output to the user (29a, 29b, 29c, 29d) and that sound signals from users (29a, 29b, 29c, 29d) of conversation groups (32a, 32b), the user (29a, 29b, 29c, 29d) is not a member of, are muted.