A hearing system adapted to be worn by a user and configured to capture sound in an environment of the user, the hearing system comprising • A sensor array (IT 11, IT 12; M1, M2; IU 1,..., IU M) of M input transducers where M ≥ 2, each for providing an electric input signal representing said sound in said environment, said input transducers p i, i =1,..., M, of said array having a known geometrical configuration relative to each other, when worn by the user, and • a detector unit for detecting movements over time of the hearing system when worn by the user, and providing location data (trac) of said sensor array at different points in time t, t =1,..., N; • a first processor (PRO1; PRO) for receiving said electric input signals and for extracting sensor array configuration specific data of said sensor array indicative of differences between a time of arrival of sound from a localized sound source S at said respective input transducers, at said different points in time t, t =1,..., N; • a second processor (PR02, PRO) configured to estimate data (doa) indicative of a location of said localized sound source S relative to the user based on corresponding values of said location data (trac) and said sensor array configuration data (tau) at said different points in time t, t =1,..., N; CHARACTERIZED IN THAT said second processor (PR02; PRO) is configured to estimate said data indicative of a location of said localized sound source S relative to the user based on the following expression for stacked residual vectors r ( S e) originating from said time instances t =1,...,N where S e represents r S e = y t ij − h ij S e R t T t e the position of said sound source in an inertial frame of reference, R t and T t e are matrices describing a rotation and a translation, respectively, of the sensor array with respect to the inertial frame at time t, and y t ij = τ ij + e t represent said sensor array configuration specific data, where τ ij represent said differences between a time of arrival of sound from said localized sound source S at said respective input transducers i, j, and e t represents measurement noise, where ( i, j) = 1,..., M, j > i, wherein h ij is a model of the time differences τ ij between each input transducer pair p i and p j. • A sensor array (IT 11, IT 12; M1, M2; IU 1,..., IU M) of M input transducers where M ≥ 2, each for providing an electric input signal representing said sound in said environment, said input transducers p i, i =1,..., M, of said array having a known geometrical configuration relative to each other, when worn by the user, and • a detector unit for detecting movements over time of the hearing system when worn by the user, and providing location data (trac) of said sensor array at different points in time t, t =1,..., N; • a first processor (PRO1; PRO) for receiving said electric input signals and for extracting sensor array configuration specific data of said sensor array indicative of differences between a time of arrival of sound from a localized sound source S at said respective input transducers, at said different points in time t, t =1,..., N; • a second processor (PR02, PRO) configured to estimate data (doa) indicative of a location of said localized sound source S relative to the user based on corresponding values of said location data (trac) and said sensor array configuration data (tau) at said different points in time t, t =1,..., N; CHARACTERIZED IN THAT said second processor (PR02; PRO) is configured to estimate said data indicative of a location of said localized sound source S relative to the user based on the following expression for stacked residual vectors r ( S e) originating from said time instances t =1,...,N where S e represents r S e = y t ij − h ij S e R t T t e the position of said sound source in an inertial frame of reference, R t and T t e are matrices describing a rotation and a translation, respectively, of the sensor array with respect to the inertial frame at time t, and y t ij = τ ij + e t represent said sensor array configuration specific data, where τ ij represent said differences between a time of arrival of sound from said localized sound source S at said respective input transducers i, j, and e t represents measurement noise, where ( i, j) = 1,..., M, j > i, wherein h ij is a model of the time differences τ ij between each input transducer pair p i and p j.