Patent Description:
An aim to process an input signal which is a mixture of multiple sources and to create an output mixture in which the relative level of the sources is modified. One example is to make the speech in a movie audio track clearer, louder, and more intelligible.

The proposed method may apply source separation to estimate the sources and remix these estimates by applying automatically generated time-varying, signal-adaptive gains. The remixing aims to fulfill a time-varying criterion concerning the separated sources and their relationship in the output mix. The output mixture has to be smooth and esthetically pleasing. For this purpose, a temporal context is taken into consideration during the generation of the remixing gains so to avoid abrupt and unaesthetic changes.

An envisioned application is to enable object-based audio personalization, e.g., based on MPEG-H Audio [<NUM>, <NUM>]. Based on MPEG Unified Speech and Audio Coding, the MPEG-H Audio standard offers many extensions for use in the context of immersive 3D audio, such as coding and rendering of multi-channel and object signals, transmission of object metadata, the compressed transmission of (speaker layout agnostic) object positions and trajectories, and it allows for personalization and user interactivity on the decoder side that is enabled and controlled by object metadata. The underlying main ideas of the new codec are to provide suitable means for an immersive experience, for universal delivery, and for personal interactivity.

Personal interactivity is a particularly demanded use case, for example, for personalizing the audio track in movies and TV programs. In fact, it has been shown that the balance between the speech and the background signals is extremely personal [<NUM>, <NUM>]. However, often, only a mono, stereo, or multi-channel mix of all sources is available instead of sub-mixes of the sources. Ways to automatically generate alternative mixes with different relative levels starting from the available mix are desired. The resulting mix has to be of high sound quality and esthetically pleasing. The system proposed in this report and shown in <FIG> can be applied for this purpose. In the example use case of object-based audio, the modules of <FIG> are located in different devices and are run in different points in time. For example, the source separation module <NUM> and the control module <NUM> and/or the temporal context module <NUM> can be located on the encoder/server side, while the remixing module is located at the decoder/end-device side.

Alternative application scenarios might involve traditional broadcasting and streaming services. In these, full personalization is usually not available (or needed), but an alternative audio track (generated as described in this report) can be generated offline and offered by the broadcasting / streaming provider. In a further envisioned application, the alternative audio track could be generated directly by the end-device. In other words, all modules are placed in the end-device.

Typically, constant gains are applied on the estimated target source and/or on the residual sources, e.g., in order to modify the SNR (signal to noise ratio) during the remixing. The SNR may be the ratio of the target signal to the at least one residual signal. These constant (over time) gains can be set by the final user, or they can be pre-defined and fixed, or they aim to optimize a global criterion. However constant gains and a global criterion have several problems:.

An alternative to constant gains, well-known among audio engineers, could be side-chain ducking, i.e., controlling the time-varying level of one (ducked) signal based on the absolute level of another ducking signal. The ducked and the ducking signals could be the outputs from the separation. This approach is also suboptimal because the amount of ducking is only based on the level of one signal and not on properties relative to all signals involved. Moreover, side-chain ducking is not robust against the unavoidable errors (e.g., leaking components) in the source separation module. Furthermore, a traditional side-chain ducking applies a stronger attenuation on the ducked signal, when the level of the ducking signal is higher. This may have a benefit in keeping the overall level of the resulting mixture approximately constant, but is not useful for, e.g., guaranteeing a level of intelligibility of a speech signal when mixed on top of a background signal. For the intelligibility, the attenuation needs to be stronger when the ducking signal is softer, so that it becomes better audible in the mixture.

<CIT> discloses a method for generating audio content. A remixing gain obtained from a variable weight proportional to energy of a separated audio source. No remixing gain is based on relative matrix comparing levels of a target signal with a level of a residual signal or of the input signal.

<CIT> discloses a method for enhanced dynamics processing of streaming audio by source separation and remixing.

<CIT> discloses a method for adaptive remixing of audio content.

<FIG>: Main concept: Given an input mix, separated source signals are estimated and remixed by applying automatically generated time-varying, signal-adaptive gains. The remixing gains are generated by the control module with the aim of fulfilling a time-varying criterion concerning the separated source signals and their relationship in the output mix. The modules in the figure can be distributed in different devices, i.e., signal encoding, transmission, and decoding can take place before or after the remixing module.

For the following explanation we can categorize all signal components in an input mixture x(t) such that they belong to one of two source signals: a target source signal s(t) (e.g., the speech recordings of all speakers in a movie soundtrack or all lead instruments in a musical recording) and a background signal b(t) comprising all residual audio sources not belonging to the target source: <MAT>.

Source separation of audio signals aims to estimate s(t), given the mixture signal x(t) (input signal <NUM>). The output of the separation is an estimate of the target source ŝ(t). Optionally more secondary sources can be estimated and output by the source separation module, e.g., an estimate of the residual sources b̂(t). It has to be noted that there are separation systems where ŝ(t) and b̂(t) do not sum up to x(t), e.g., [<NUM>], but an estimate for b̂(t) can also be obtained as b̂(t) = x(t) - ŝ(t). In examples below, even if either ŝ(t) (or, respectively, x(t)) is processed, it is also possible to obtain an estimation of x(t)(or, respectively, ŝ(t)), simply by adopting the formula b̂(t) = x(t) - ŝ(t) (x(t) = ŝ(t) - b̂(t), respectively).

A post-filtering can be applied to ŝ(t) and b̂(t), e.g., an equalizer for enhancing and/or attenuating certain frequency regions or a post-processing for removing musical noise.

In many application scenarios, the estimated source signals ŝ(t) and b̂(t) are not intended to be listened to separately, but they are remixed with a partial modification of the relative levels [<NUM>, <NUM>]. The notion of Signal-to-Noise Ratio (SNR) can be used here, referring the level difference between s(t) and b(t) or their estimates.

There are many solutions for source separation, e.g., [<NUM>, <NUM>, <NUM>, <NUM>] and references therein. The solutions may rely on hand-designed audio signal processing algorithms, e.g., [<NUM>], also referred to as "classical signal processing", or the solutions may be based on deep learning, see e.g., [<NUM>, <NUM>]. The technique proposed in this report is not limited to any specific source separation system. The estimates of the sources are in real world likely not perfect. Various imperfections, such as cross-leaking components, artifacts, distortions, and colorations can be introduced by the source separation. It is important to consider this fact while remixing.

<FIG> shows an example of system <NUM>. The system <NUM> permits signal-adaptive remixing of separated audio sources. The system <NUM> processes an input signal <NUM> (input mix) x(t). The input signal may be a mono signal. This may apply to the target signal and the residual signal. The system <NUM> provides, for example, an output signal (output mix) y(t) <NUM> (further post-processing can be applied, such as loudness normalization, dynamic range compression, or applying equalization). The system <NUM> may include a source separation block <NUM>. The source separation block <NUM> may extract different signals from the input signal <NUM> (e.g., by signal processing, filtering, etc.). For example, from the input signal <NUM> a target signal <NUM> may be separated from at least one residual signal <NUM>. An example may be, for example, a target signal ŝ(t), which is separated from a background signal b̂(t) (residual signal). For example, the target signal <NUM> may be a speech, while the background signal <NUM> may include other sounds present in the input signal (e.g. ambience, effects, and music). In other cases, the target signal may be a signal which is filtered from the input signal <NUM>, because maybe a user intends to have an increased level for the target signal <NUM> in respect to that at least one residual signal <NUM>. For example, the target signal <NUM> may be speech only, estimated by blind source separation, and so on. It is possible for a user to identify the target signal <NUM> to be separated from the residual signals <NUM>. A remixing block <NUM> may be provided, to provide the output signal (output mix) y(t) <NUM>. The remixing block <NUM> may be input with the target signal <NUM> and the one or more residual signals <NUM> and can remix them according to modified gains <NUM>. The remixing block may therefore operate by using a remixing matrix with coefficients (gains <NUM>) which, in general, vary in time. It will be subsequently explained that at least one gain <NUM> at the remixing block <NUM> is variable in time: e.g., different time instants or time slots of the target signal ŝ(t) (<NUM>) and/or the at least one residual signal b̂(t) (<NUM>) are subjected to gains which vary along the elapsing of time, and in particular based on the values (and for metrics obtained from them) of the target signal ŝ(t) (<NUM>) at different (e.g., future or past) time instants or time slots. In fact the remixing gains are modified in such a way that they evolve with time and they can, for example, provide some particular functions. Functions will subsequently be discussed (e.g. as smoothing gains) for reducing the level of the background in respect to the level of speech (e.g., embodying a function which is normally performed by the so-called ducking functions).

A control block <NUM> is provided, which has, in input, the target signal <NUM> and, either the input signal <NUM> and/or the one or more residual signals <NUM> (the input signal <NUM> or the target signal <NUM> is also called "signal <NUM>" or "first signal <NUM>"). (<FIG> shows both the input signal <NUM> and the background signal <NUM> being input to the control block <NUM>, but in some examples it may be that only one of the input signal <NUM> and the background signal <NUM> is actually inputted onto the control block <NUM>). The control block <NUM> makes use of a temporal context block <NUM>. The control block <NUM> may request temporal context information <NUM> by exerting a control. The control block <NUM> provides temporal information <NUM> on the current time instant or time slot which will be subsequently used as temporal context information <NUM> (e.g., for subsequent time instants or time slots, and/or for refining a previously obtained rough gain <NUM>, so as to deviate from the rough gain <NUM> to obtain the remixing gain <NUM>). As it will be shown later, the temporal information <NUM> on the current time instant or time slot may include at least one of the utterance integration block <NUM> (e.g., <NUM>, <NUM>) or information associated thereto; rough gain <NUM> and/or activity information (e.g. gate information) <NUM>; a gated gain (e.g., rough gain <NUM>, e.g. <NUM>); and the at least one remixing gain <NUM> (e.g., gsmooth(t-<NUM>)). Some of these information will be explained in greater detail below.

On the basis of the temporal context information <NUM>, the control block <NUM> appropriately defines, time instant by time instant or time slot by time slot, the at least one remixing gain (e.g. remixing gains) <NUM> to be provided to the remixing block <NUM>. Accordingly, the obtained output signal (output mix) <NUM> will be remixed by keeping into consideration not only the target signal <NUM> at a particular time instant or time slot, but also on the target signal in the temporal context (e.g., future or past time instants or slots).

The input signal <NUM> and the separated signals <NUM>, <NUM> (<NUM>), and/or the processed versions of those signals, are signals evolving in time along a discrete succession of time instants or time slots. Each time instant may be, for example, associated to a particular sample (e.g., signal in the time domain), e.g. present in the input signal (e.g. ambience, effects, and music). Otherwise, time may be understood as being subdivided (e.g. partitioned) into a plurality of time slots, and each time slot may be associated to a signal description in the frequency domain (e.g., digital Fourier transforms DFT, short-time Fourier transform STFT, fast Fourier transform FFT, and so on). In the frequency domain, a plurality of values may be associated to the particular time slot, each value being, for example, a coefficient to be associated to a particular frequency. There is no particular difference in this case between whether the signal(s) is(are) in the time domain or in the frequency domain. Hence, most of the following explanations are common for both the time domain case and the frequency domain case.

<FIG> shows an example of the evolution in time of the target signal <NUM> and the residual signal <NUM> or input signal <NUM> (signal <NUM>, or "first signal <NUM>", is used for indicating either the residual signal <NUM> or the input signal <NUM>).

As seen in particular in <FIG>, a time evolution is shown as a typical horizontal line, where time instants or time slots t are along a discrete succession. For each time instant or time slot in the discrete succession, both the target signal <NUM> and the signal <NUM> (<NUM>, <NUM>) presents a value either in the time domain or in the frequency domain (the value may have multiple components; for example, if the value is in the frequency domain, a plurality of components may be provided, e.g. one component for each frequency band). For example, at the time slot <NUM> (subsequently often indicated as "current time instant or time slot <NUM>" or "current determined time instant or time slot <NUM>" or "determined current time instant or time slot <NUM>"), the target signal <NUM> (or processed version thereof) presents the value <NUM>, and the signal <NUM> (<NUM>, <NUM>) (or processed version thereof) presents the value <NUM>. Reference signs <NUM> and <NUM> refer to windows of consecutive time instants or time slots which are subsequent, in the discrete succession, to the current time instant or time slot <NUM>. Analogously, reference signs <NUM>, <NUM> refer to windows of consecutive time instants or time slots which are before, in the discrete succession, the current time instant or time slot <NUM>. Reference sign <NUM> refers to the time instant or time slot immediately before the determined current time instant or time slot <NUM> (where the determined current time instant or time slot <NUM> is expressed as t, the time instant or time slot immediately before the determined current time instant or time slot <NUM> is expressed as t-<NUM>). In some examples, the at least one remixing gain is first obtained for the determined current time instant or time slot <NUM>, and subsequently the determined current time instant or time slot <NUM> is obtained. The target signal <NUM> and the signal <NUM> (<NUM>, <NUM>) (or processed versions thereof) also present some values for the slots of the windows <NUM> and <NUM>, even though they are not shown in <FIG>. Put together, in some examples the windows <NUM> and <NUM> and the determined current time instant or time slot <NUM> may form a time window which includes the determined current time instant or time slot <NUM>. In accordance to the temporal context information as required, it is possible to make use of any of the time slots or time instants <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, which are all in the future or in the past. In some cases, the windows in the past, in the future, or both in the past and future may have a predetermined time length (e.g., a predetermined number of time instants or time slots). For example, window <NUM> may comprise a predetermined number of time instants or time slots. It may be, in some examples, that the plurality of time instants or time slots <NUM> are some slots within the window <NUM>. In some examples, at least one (or both) of the windows <NUM>, <NUM> is immediately before or immediately after the current time instant or time slot <NUM>. In some examples, at least one (or both) of the windows <NUM>, <NUM> is not immediately before or immediately after the current time instant or time slot <NUM>.

The time instant or time slot <NUM> (subsequently indicated to as "future time instant or time slot <NUM>") happens to be, in the time evolution (according to the discrete succession) of the target signal <NUM> and of the signal <NUM>, subsequent to the current time instant or time slot <NUM>. Accordingly, the time instant or time slot <NUM> is understood to be "in the future" with respect to the time instant or time slot <NUM>. The values of the target signal <NUM> and of the signal <NUM> (<NUM>, <NUM>), or processed versions thereof, are respectively indicated to with <NUM> and <NUM>. It will be shown that it is possible to have different remixing gains <NUM> for different time slots or time instants. Moreover, it is possible to adapt the gain <NUM> associated to the time instant or time slot <NUM> as being obtained by also considering the value of the time instant or time slot <NUM>.

The same may be performed for other time instants or time slots with respect to the time instant or time slot <NUM>.

However, when the time instant <NUM> is processed, the values <NUM> and <NUM> at the future time instant or time slot <NUM> may be already known (e.g., stored in buffers). Below, where it is explained that the time instant or time slot <NUM> is the current time instant or time slot, it is meant that the current time instant or time slot <NUM> is currently processed, even though the future time instants or time slots (e.g., <NUM>) are already known and/or some form of preprocessing is already performed to the future time instants or time slots, e.g., <NUM>. Accordingly, the fact that some time instants or time slots are in the future shall not be understood as obtaining some features which are unknown, but it is more than the current time instant or time slot <NUM> is adapted to the future time instant or time slot <NUM>, which is already known.

For example, it is possible to first obtain rough remixing gains (e.g. according to determined remixing criteria) for a plurality of time instants or time slots (e.g. for all the temporal evolution of the input signal), including any of <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. After having obtained the rough remixing gains <NUM>, it is subsequently possible to obtain the remixing gains <NUM>, by performing deviations from the rough remixing gains <NUM> (in particular in transitory intervals or transition intervals) so as to obtain the remixing gains <NUM>, e.g. by making use of temporal context information. This process may be performed iteratively, e.g. by first obtaining the remixing gains <NUM> for time instants in the past, then for a present time instant, and subsequently for the time instants in the future.

Moreover, it is intended that, after having processed the current time instant or time slot <NUM> (e.g. t<NUM>), subsequently the current time instant or time slot <NUM> is updated to another time instant or time slot (e.g. the time instant or time slot immediately subsequent t<NUM>=t<NUM>+<NUM>).

<FIG> also shows that the control block <NUM> receives (or measures) metrics from the current time instant or time slot <NUM>, while the temporal context block <NUM> receives (or measures) metrics taken from the values <NUM> and <NUM> of the target signal <NUM> and of the signal <NUM> (residual signal <NUM> or input signal <NUM>, or processed version thereof) at the future time instant or time slot <NUM> (the same could be done for a past time instant or time slot, which are not shown in <FIG> but which may be used exactly as the future time instant or time slot <NUM>). The same may apply to the time instants or time slots <NUM> (in the past) or the time instants of the window <NUM> (in the future).

The metrics which are obtained may include, for example, absolute metrics <NUM> and/or <NUM> associated to absolute magnitudes (e.g., loudness, level, power, energy, etc., of the particular signal <NUM> or <NUM>) at the current time instant or time slot <NUM> or <NUM>, as can be obtained, for example, from the value <NUM> and/or <NUM>.

The metrics which are obtained include relative metrics <NUM> and <NUM>. The relative metrics <NUM> and <NUM> may be the at least one metrics. For example, a relative metrics <NUM> is obtained by comparing an absolute metrics of the target signal <NUM> (e.g., as obtained from value <NUM>) with an absolute metrics of signal <NUM> (<NUM>, <NUM>) at the current time instant slot <NUM> (e.g., as obtained from value <NUM>). Another relative metrics <NUM> takes into account values <NUM> and <NUM> of the target signal <NUM> and the signal <NUM> (<NUM> or <NUM>) in the at least one future and/or past time instant or time slot <NUM>. The metrics <NUM> and <NUM> are shown as being obtained at comparing blocks <NUM>' and <NUM>', respectively. An example of relative metrics <NUM>, <NUM> the (possibly frequency-weighted) may be the relative intensity of the signals, e.g., SNR(t) (also indicated as SNRin(t) in formulas (<NUM>) and (<NUM>)), which may imply, for example, a ratio between absolute metrics such as those above. Multiple relative metrics may form a composite relative metrics. A metrics may imply, for example, a norm on the instant value of the signal. For example, a <NUM>-norm, a <NUM>-norm, etc. may be used. The metrics may be a norm, such as <NUM>-norm, a <NUM>-norm, etc. A norm may provide a non-negative real number which keeps into consideration the channels of the signal (e.g., the sum of their absolute values, the square root of the sum of their squared values, etc.). Further, multiple metrics (absolute metrics, relative metrics, or both) may be combined with each other to obtain a metrics which is a composite metrics (and partially relative metrics and partially absolute metrics). An example of absolute metrics <NUM>, <NUM> is the absolute intensity of the signals, possibly frequency-weighted, e.g. absolute metrics such as the intensity of the target signal <NUM> and/or the intensity of the signal <NUM> (e.g., <NUM>, <NUM>), respectively. Another example of absolute metrics <NUM>, <NUM> may be an estimate of the perceived time-varying loudness and/or loudness difference. Another example of absolute metrics is a time-dependent quality or intelligibility metric or a speech activity probability. Another example of absolute metrics is a combination of these or other time-dependent features of the signals (multiple absolute metrics may form a composite absolute metrics).

Particular functions may be obtained with the present examples. For example, it is possible to apply the most appropriate remixing gains at each time instant or time slot (<NUM>, <NUM>, etc.) and, for example, smoothing some gains (e.g., when transitioning from a first remixing gain to a second remixing gain, as will be explained below).

The generation of the at least one remixing gain <NUM> may be subjected to the definition of one or more remixing criteria. A remixing criterion may be, for example, a criterion for obtaining a particular goal (e.g., attenuating a background signal or boosting a particular target signal). The choice of a particular criterion may generally be associated to the metrics <NUM> and/or <NUM> (or respectively <NUM> and <NUM>) in a particular time slot <NUM> (or respectively <NUM>). A remixing criterion may therefore be associated to the value of a particular time instant or time slot <NUM> or <NUM>. It may be seen that, in some cases, the current time instant or time slot <NUM> and the past and/or future time instant or time slot <NUM> are two time instants or time slots for which different remixing criteria are chosen (e.g. due to different results of an activity detection operation). It may be that, for the determined current time instant or time slot <NUM>, the control block <NUM> chooses not to completely follow the remixing criterion as would be defined based on the metrics <NUM> and <NUM> of the target signal <NUM> at the current time instant or time slot <NUM>: the control block <NUM> may therefore keep into account the temporal context <NUM>. For example, while different remixing criteria may be defined for the current time instant or time slots <NUM> and <NUM> on the basis of the metrics <NUM> and <NUM> associated to the same time instant or time slot, the remixing criteria can also be not completely respected, by virtue of using the temporal context <NUM> and in particular, the metrics <NUM> and/or <NUM> associated to future and/or past time instants or time slots, thereby operating a deviation.

<FIG> shows an example of operation which may be obtained through the examples above. Here, it is possible to see that the target signal <NUM> (which could be imagined to be a human voice) is to be remixed with respect to noise (residual signal <NUM>). The speech, when present, is at a loudness level LV. The noise <NUM> (residual signal, background signal) is shown to be acquired as having a constant level LH1. At time instant tB, speech <NUM> starts. The speech <NUM> transitorily ends at time instant tF, but restarts again at instant tL, hence defining a brief time interval <NUM> without voice <NUM> (it may be a time interval between the enunciation of one first word and the enunciation of one second word). Subsequently, at instant tK (also indicated as tE), the speech <NUM> ends again (it may be that the speaker does not enunciate words anymore).

At instant tB, noise <NUM> (residual signal, background signal), which was previously at level LH1, is to be subsequently played back at level LH2 < LH1, so as to increase the quality of the output signal <NUM> by reducing the noise <NUM> (by a quantity indicated by <NUM> in <FIG>), to permit the listener to better understand the speech <NUM>.

In theory, for the time instants or time slots before time instant tB, a unitary remixing gain (e.g. <NUM> dB) could be applied to the noise <NUM>, while a remixing gain less than unitary (negative in decibel) could be chosen for time instants or time slots after time instant tB (in particular in the interval tDA). Hence, the level of the noise <NUM> would be modified from level LH1 to a level LH2 which causes the difference between the speech <NUM> and the noise <NUM> to be the quantity indicated with <NUM> (clearance). This is a behavior which is subdivided in two remixing criteria:.

An example in formula (<NUM>) (see below, and see also formula (<NUM>)).

The first remixing criterion may be based, for each time instant or time slot before tB, on relative and/or absolute metrics <NUM>, <NUM> associated to exactly that time instant or time slot. On the other side, the second remixing criterion may be based, for each time instant or time slot in the interval tDS (but which is in the future with respect to the time instants or time slots before tB), on relative and/or absolute metrics <NUM>, <NUM> associated to exactly that future time instant or time slot. At time slot or time instant tB, an abrupt change of criterion (and of gate, accordingly) would occur, and the noise <NUM> would jump from level LH2 to level LH1.

However, it has been understood that this abrupt change would not be pleasant for a listener, and could cause an unwanted pumping effect.

A more smoothed transition (e.g. identified by ramp <NUM> in <FIG>) is therefore in principle preferable. As show in <FIG>, starting from time instant tA < tB, a gradual reduction of the remixing gain for the noise <NUM> is performed. Accordingly, the pumping effect is not audible or at least less audible. Therefore, throughout the time interval tDS, the gain for the background <NUM> (residual signal) is progressively reduced in respect to the level LV of the speech (target signal <NUM>).

Notably, we obtain a subdivision into three regions:.

In the second, intermediate region <NUM> (interval tDS), the determined current time instant or time slot <NUM> will have a remixing gain <NUM> which is intermediate between those associated to the current time instant or time slot before tA and after tB.

The same applies in the interval tDR (in which ramp <NUM> is experienced), at which the remixing gain <NUM> also changes gradually again causing the noise <NUM> to change from level LH2 to a higher level LH1. Even in this case (ramp <NUM>), at any determined current time instant or time slot before tE (e.g. in interval tOR), the remixing criterion provides a rough value of the gain that would cause the level LH2, while a time instants in the time interval IDR after tE should have a remixing gain causing the level LH1. However, it is possible to take into account the gain as it would be at in the time instants after tER according to the criterion, and accordingly, choose a remixing gain value intermediate between the gain value that causes the level LH2 and the gain value which causes the gain LH1. This is dual to the above-mentioned case of ramp <NUM>, where at any determined current time instant or time slot before tB (e.g. in interval tOA), the remixing criterion provides a rough value of the gain that would cause the level LH1, while a time instants in the time interval tDS after tB should have a remixing gain which is the gain that causes the level LH2. However, it is possible to take into account the gain as it would be at in the time instants after tEA according to the criterion, and accordingly, choose a remixing gain value intermediate between the gain value that causes the level LH2 and the gain value which causes the gain LH1. The duality can be easily seen in intervals tDS and tDR, and is obtained, for example, by applying formulas (<NUM>), (<NUM>), and (<NUM>) (see below). To achieve this goal, in one example it is possible to apply the shifting as discussed, for example, in <NUM>. Other techniques are notwithstanding possible.

In time interval <NUM>, there is no gradual modification between two different remixing criteria, but instead it is remained in the remixing gain as would be defined by the second remixing criterion instead of moving towards the gain defined by the first remixing criterion. It is possible to make use, in some cases, of an utterance integration <NUM>, which permits to recognize that the time interval <NUM> between the two time intervals (e.g. encompassing tDA and tOR) at which the speech is obtained is still an interval in which the target signal <NUM> is active. It is noted that some remixing criteria may be dominant over other remixing criteria. For example, the second remixing criterion adopted in the remixing region 200H2 is dominant over the first remixing criterion adopted in the remixing region 200H1: we want to maintain the gain <NUM> for the residual signal <NUM> low for coping with situations in which the absence of the target signal is only due to a pause within two words, without increasing the loudness of the noise <NUM>. To the contrary, the first remixing criterion is non-dominant: in the intermediate region <NUM>, the ramp <NUM> is immediately generated, without waiting too much. Hence, before moving from one dominant remixing criterion towards a non-dominant remixing criterion, there may be inspected, in the target information in a time window immediately after the determined current time instant <NUM>, whether the totality (or at least a great number, greater than a first predetermined threshold) of future time instants <NUM> (or <NUM>) are associated to the non-dominant remixing criterion; while before moving from one non-dominant remixing criterion towards a dominant remixing criterion, there may be no such inspection, or in alternative there may be a less strong condition than that for transitioning from the dominant criterion towards the non-dominant criterion: for example, when transitioning from the non-dominant remixing criterion to the dominant remixing criterion there may be inspected whether a little number of future time instants (e.g. over a second predetermined threshold) or time slots is associated to the dominant criterion, wherein the second predetermined threshold is lower than the first predetermined threshold.

By virtue of the above, it is possible to see that a first remixing criterion and a second remixing criterion may be, in general, used for generating at least one rough remixing gain (e.g. in non-transitory phases). The rough remixing gain <NUM> may subsequently be corrected by applying a deviation (see also below), e.g. in transitory phases.

The different remixing criteria apply different gains (e.g. different rough gains) and, therefore, will cause different remixings. The discrimination between the remixing criteria is generally made based on a criterion condition. The criterion condition may take into account the metrics <NUM> (absolute metrics for the determined current time instant <NUM>) and/or <NUM> (relative metrics determined current time instant <NUM>) (see <FIG>). Therefore, if different time instants or time slots have different values <NUM>, and consequently different metrics <NUM> and/or <NUM>, it may happen that they end being associated to different remixing criteria.

The criterion condition may take into account the metrics <NUM> (absolute metrics for the determined current time instant <NUM>) and/or <NUM> (relative metrics determined current time instant <NUM>) on the target signal <NUM> or a processed version thereof (such as version <NUM>, <NUM> and, e.g. in case of relative metrics <NUM>, also versions <NUM> and <NUM> of the input mix <NUM> or the residual signal <NUM>).

In non-transitory conditions (such as in high gain region 200H1 and in the low gain region 200H2), the first and second criteria may be easily respected. For example, in the high gain region 200H1, the first remixing criterion is respected: the gain for the time instants or time slots of the background signal <NUM> is maintained unitary. The second remixing criterion may provide a reduction of the gain for the residual signal <NUM> with respect to the first remixing criterion (or in particular, an increase of the ratio between the remixing gain associated the remixing gain associated to the target signal over to the residual signal <NUM> from the first remixing criterion to the second remixing criterion).

Notwithstanding, in some cases, as explained above, it is possible to deviate from the first and second criteria (e.g., in case of transitory; see intermediate region <NUM> in <FIG>). An example is provided in the intermediate region <NUM>, in which the ramp <NUM> is generated and the gains for the residual signal <NUM> are progressively reduced, to reach the reduced gain prescribed by the second remixing criterion for increasing the distance from the target signal <NUM>. Notably, the deviation may be based on the temporal context information <NUM>. Of course, the example of <FIG> is very general (see also formula (<NUM>) below), but other different criteria and/or criterion conditions may be chosen.

It is also to be noted that each of the first and second remixing criterion is associated to a rough remixing gain (the rough remixing gain based on the first remixing criterion being in principle different from the rough remixing gain based on the second remixing criterion), which can be, notwithstanding, modified (e.g. corrected, deviated). The deviation may be based, for example, on the temporal context information <NUM>. The deviation is evident in <FIG> by virtue of the ramp <NUM>: before the time instant tB the first criterion would prescribe a higher gain for the residual signal <NUM>, while, after tB the second criterion would imply that the gain should be at a lower level. By virtue of the deviation, the ramp <NUM> is advantageously obtained. The same applies to ramp <NUM>: before the time instant tE the second remixing criterion would prescribe a lower gain for the residual signal <NUM>, while, after tE the first remixing criterion would imply that the gain should be at a higher level. By virtue of the deviation, the ramp <NUM> is advantageously obtained.

In particular, the deviation may take into consideration the time slots or time instants which are immediately subsequent to the determined current time instant or time slot <NUM> (e.g. window <NUM> or <NUM> in <FIG> and <FIG>). Alternatively or in addition, the temporal context information <NUM> used for the deviation may be based on a remixing gain obtained for a previous slot or instant (e.g., time instant <NUM> or slot <NUM> in <FIG>), which may be at least one of the time slot or instant immediately preceding the determined current time slot <NUM>. Accordingly, the deviation may be based on a linear combination of the rough gain <NUM> as obtained for the determined time instant or time slot <NUM>, and the previously obtained remixing gain (also indicated with gsmooth(t-<NUM>)) of the immediately preceding time slot or time instant <NUM>. An example is provided in formulas (<NUM>), (<NUM>), and (<NUM>) (see below).

Some transient variation of the target signal <NUM> and/or the residual signal <NUM> or input signal <NUM> may cause a time instant or slot to be associated to have an incorrect value, so that its metrics <NUM> (absolute metrics) or <NUM> (relative metrics) may be incorrect, which could drive to be associated to a wrong remixing criterion. In addition or alternatively, there may be the possibility of having a transient disturbance, noise. It is also possible to experience a pause between two words: in <FIG>, during the time interval <NUM>, the time instants or time slots appear to be associated to the first criterion (no ducking, like in the region 200H1), and the gain <NUM> for the residual signal <NUM> should move towards the high gain. This means that the listener should experience, after time instant tF the loudness of the residual signal <NUM> gradually increasing. This would cause an unpleasant audible effect. To cope with this problem, in interval <NUM> it is possible to make use of temporal content information <NUM> regarding the future time slots or time instants, so as to conclude that the first remixing criterion that would appear from the metrics is only temporary, and the first remixing criterion will be used soon. Accordingly, it is possible to deviate from the first remixing criterion (which would cause the increase of the gain for the residual signal <NUM>) by performing a deviation that maintains the gain constant.

It is possible, as explained above, to verify whether a deviation condition is fulfilled or not. The deviation condition may be at least partially based on the temporal context information <NUM> (e.g., a window <NUM> or <NUM> of time instants or time slots, which are in the future with respect to the determined time instant or time slot <NUM>). If all the future instants are associated to the second criterion (provided that they are in a time window <NUM> or <NUM> of a predetermined length, also indicated with THOLDAHEAD), then the deviation is performed by correcting the rough gain <NUM>. Accordingly, the gain <NUM> for the residual signal <NUM> may gradually increase (time interval tDR).

An example valid for example for the transitory in time interval tDR and in time interval <NUM> is provided by method <NUM> of <FIG>. At step <NUM>, it may be determined whether the determined current time instant or time slot <NUM> is on the first or second remixing criterion. This may be an example of the evaluation of the criterion condition discussed above. This may be based on metrics <NUM> (absolute metrics, e.g. intensity, etc.) and/or <NUM> (relative metrics, e.g. SNRi, etc.) on the determined current sample or time instant. Subsequently, a rough gain is generated at step <NUM> according to the determined criterion. Accordingly, the first and second criteria may prescribe different gains <NUM>.

Subsequently, at step <NUM>, it is intended to see whether the rough remixing gain <NUM> is to be corrected. Therefore, temporal context information may be obtained from the temporal context block <NUM>. At step <NUM>, the deviation condition is evaluated. A condition on the immediately subsequent time instants or time slots <NUM> or <NUM> immediately subsequent to the determined current time instant or time slot <NUM> may be evaluated. For example, if, within a predetermined time window of a predetermined length, all the subsequent time instants or time slots are associated to the different criteria, then it is transitioned to step <NUM>, in which the deviation is performed by correcting the rough gain, e.g. using the techniques discussed with respect to formulas (<NUM>) and/or (<NUM>). This may be obtained, for example, by defining the at least one gain <NUM> as a linear combination which keeps into account both the rough gain (g(t)) as obtained from the metrics <NUM> (absolute metrics) and <NUM> (relative metrics) on the target signal <NUM> at the determined time instant or time slot <NUM> and by also taking into account the preceding version (e.g. gsmooth(t)) of the at least one gain <NUM> immediately preceding the determined current time instant or time slot <NUM>. Accordingly, it may be gradually transitioned from a particular criterion to another criterion.

In case the evaluation of the deviation condition at step <NUM> determines that not all the future instants in the future time window <NUM> or <NUM> will be associated to another criterion, but some of them will also be associated to the current criterion as determined at step <NUM>, then it is transitioned towards step <NUM> and the gain <NUM> is maintained constant with respect to the previous one (i.e., the gain <NUM> as already obtained for the immediately previous current time instant or time slot <NUM> immediately preceding the determined time instant or time slot <NUM>). More in general, at step <NUM> it is possible to take into account a time instant or time slot preceding the time instants or time slots (e.g. <NUM> or <NUM>) following the determined time instant or time slot, such as one of the two time instants or time slots (t and t-<NUM>) immediately preceding the time instants or time slots (e.g. <NUM> or <NUM>) in the time window following the determined time instant or time slot, so as to compare whether the criterion associated to the future time window <NUM> or <NUM> is the same of the criterion associated to the one time instant or time slot (t, t-<NUM>) immediately preceding the time instants or time slots (e.g. <NUM> or <NUM>) following the determined time instant or time slot, while at step <NUM> there may be, in addition or in alternative, determined the remixing criterion of the time instant or time slot t-<NUM>, as well.

In the present example, one remixing criterion is dominant (prevailing) with respect to another remixing criterion. For example, in <FIG> the second remixing criterion is dominant with respect to the first remixing criterion: while in time interval <NUM> the gain of the background signal <NUM> is maintained low, the same is not carried out for time interval tDS (the ramp <NUM> starts quickly). The second remixing criterion prevails over the first remixing criterion because we want that a quick pause between two words (between time instants tF and tL) has not a change in the gain <NUM> for the background signal <NUM>. This is not the situation occurring when transitioning in the interval tos, where there is a transition from the first remixing criterion to the second remixing criterion: we want that as soon as a speech starts (e.g., in instant tB) the gain of the background signal <NUM> is quickly reduced (despite gradually). Hence, the second remixing criterion is chosen as being dominant with respect to the first remixing criterion. This also permits to avoid the evaluation of the deviation condition <NUM> and the subsequently use of block <NUM> when transitioning from the first remixing criterion to a second remixing criterion (interval tDS in <FIG>). Hence, a version of <FIG> for a transition from a non-dominant criterion to a dominant criterion (like in tDS) would only imply blocks <NUM>, <NUM>, <NUM>, and <NUM>, while blocks <NUM> and <NUM> would be directly connected without evaluations of other conditions. Blocks <NUM> and <NUM> would be deactivated.

<FIG> shows a variant <NUM>, which is not only valid for transitories (e.g. at transitions). This variant <NUM> is also valid for the non-transition regions (e.g., region 200H1 and region 200H2 in <FIG>). Here, method <NUM> may have blocks <NUM>, <NUM> and <NUM>, which may be the same as those of method <NUM> of <FIG>, or one of its variants, some of which are discussed above and below. However, a preliminary condition is evaluated in block <NUM>, in which it is evaluated whether all the future instants or slots of the window <NUM> or <NUM> (e.g. immediately after the determined current time instant for slot <NUM>) will be associated to the same criterion that has been determined in step <NUM>. If the future instant time slots <NUM> or <NUM> are associated to the same criterion that is chosen for the determined current time instant or time slot <NUM> (or, in some variants, to the immediately preceding time instant or time slot <NUM>, t-<NUM>) at step <NUM>, then it is transitioned to step <NUM>, where the same criterion is used and the rough gain is used as the determined gain without deviations. If, to the contrary, the evaluation of the preliminary condition <NUM> is negative (and it is therefore understood that there are, subsequently, some time instants or time slots for which the criterion will be different from that chosen at step <NUM> for the determined current instant or time slot <NUM>), then the deviation condition <NUM> is evaluated. At that point, the same outcomes of method <NUM> of <FIG> and the same consequences (e.g., blocks <NUM> and <NUM>) are followed. As explained above, the blocks <NUM> and <NUM> may, in some examples, be avoided in the case in which the criterion determined at step <NUM> is not a dominant criterion (in those cases in which a dominant criterion is actually defined). Method <NUM> may therefore describe the operations of <FIG> in such a way that the non-transitory time intervals (e.g., high gain region 200H1 and low gain region 200H2 in <FIG>) are controlled by block <NUM>.

In some examples, method(s) <NUM> and/or <NUM> may include, e.g. at the end, shifting the least one gain (<NUM>, gsmooth) as obtained for each time instant or time step of the discrete succession of time instants or time slots by a predetermined number of time instants or time steps towards the past.

<FIG> shows an example <NUM> that explains how to operate, in particular, for performing the deviation and/or for performing the evaluation. It is also further discussed and explained in subsection <NUM> herein below. Here, we see a gain evolution in time. The evolution shows the determined current time instant <NUM> (time t) the time instant or time slot <NUM> and, immediately subsequently to the determined current time instant <NUM> (t), a window <NUM>, <NUM> of rough gains <NUM> is also defined. The window also subsequently explained as "tholdahead" is defined. The window may have a predetermined length.

Notably, before the determined current time instant (t), the gain(s) <NUM> (including the immediately preceding time instant or time slot <NUM> or t-<NUM>) is(are) the gain(s) as already obtained (e.g., correct gains in previous iterations for preceding time instants, e.g. gsmooth(t)). On the other side, the remaining instants (instant or slot <NUM> and the subsequent ones) may have only the rough gain(s) <NUM>, previously obtained based on the metrics (absolute metrics and/or relative metrics) on the target signal <NUM> that are at those time instants. Therefore, during the process, the final gain(s) <NUM> of each (all) time instant(s) are subsequently and iteratively updated.

In order to take into consideration the temporal context (e.g. at step <NUM>), an evaluation may be performed on the window <NUM> or <NUM> (tholdahead) of the immediately subsequent time instants or slots. Here, the rough gains <NUM> (g) are evaluated. It is looked (determined) whether they are associated to the first criterion or the second criterion, and/or it is looked (determine) whether they have the same remixing criterion of one of the time instants or slots immediately preceding the window <NUM> or <NUM>, e.g. the determined time instant or slot <NUM>. This may be the evaluation which is carried out in step <NUM> of <FIG> and <FIG>, and that causes the transitioning towards either step <NUM> or step <NUM>. A discussion will be performed in subsection <NUM>.

It is to be noted that it is not strictly necessary to evaluate the obtained gains in the window of rough gains. It is also simply possible to evaluate whether the first or second evaluation criterion are chosen (e.g., roughly chosen). After that, the correction will be performed as explained above.

<FIG> shows an example of control block <NUM>, which may be adopted in some cases (e.g. it may cause the operations like in <FIG>). However, in some examples the system of <FIG> may be different from the block <NUM>. In this case, as input to the control block <NUM> there are provided the separated target source <NUM> or ŝ(t) and the input signal (input mix) x(t) <NUM>, which is here considered the so-called first signal <NUM>. As an alternative to the provision of the input signal <NUM>, it would also be possible to provide at least one of the residual signals b̂(t) <NUM> as signal <NUM>.

Notwithstanding, the description is here based by mainly assuming it is the input signal <NUM>, which is provided to the control block <NUM>. It will be shown that the control block <NUM> provides remixing gain gsmooth(t) <NUM> which are to be provided to the remixing block <NUM>.

Both the target signal <NUM> and the first signal <NUM> (<NUM>, <NUM>) may be processed to obtain a short-term level estimation <NUM> and <NUM>, respectively. The operations of the short-term level estimations will be explained below in subsection <NUM>, but it is already explained that they are associated to a first order IIR filter. A smoothing time constant α may be used for both blocks <NUM> and <NUM>. It is also possible to transfer into a logarithmic domain to better reflect the magnitude response of the human audio.

On the signals <NUM> and <NUM> (<NUM>, <NUM>) (or on their processed versions <NUM> and <NUM>) it is possible to perform a first target activity detection at TAD block <NUM>. The operations of the TAD block <NUM> are also discussed below in detail in block <NUM> and in formula (<NUM>). In an example, the TAD block <NUM> may compare the target signal <NUM> (or a processed version <NUM> thereof) with an absolute threshold <NUM> ("absolute gate") and/or can compare the target signal <NUM> (or a processed version <NUM> thereof) with a relative threshold <NUM> ( "relative gate") (e.g., in comparison with the first signal <NUM>, i.e. the input signal <NUM> or one of the residual signals <NUM> or a processed version <NUM> thereof). If the target signal <NUM> is not big enough in comparison with the first signal <NUM> (input signal <NUM> or the residual signal(s) <NUM>), then it is imagined that in the particular time instant or time slot, the target signal <NUM> is inactive. Accordingly, in short term activity information <NUM> may be generated indicating that the target signal is active. If, on the other side, the target signal <NUM> is not big enough (e.g., either in absolute terms or in relative terms with respect to the input signal or one of the residual signals) then the short-term activity information <NUM> indicates that the target signal <NUM> is supposed to be inactive (non-active). Here, the short-term activity information <NUM> is considered to be a gate signal, which may be understood as a binary information, which indicate that the target signal <NUM> is considered to be active or non-active. It is to be noted that the short term activity detection information <NUM> is not definitive in at least some examples. In fact, downstream, this information may be filtered and changed by also taking into account the behavior of the target signal <NUM> for the time instants and/or time slots closely consecutive to the determined current time instant.

It is to be noted that the short term activity detection information <NUM> may in general take into account uniquely the evolution of the signals <NUM> and <NUM> (e.g., <NUM> or <NUM>) of the processed versions thereof <NUM> and <NUM>, but in general does not take into consideration the signal (e.g. <NUM> and/or <NUM>) at samples and/or instants around the considered time instant. As it will be shown in the following, this can give some issues, since it is possible that a pause is performed between two different words in a speech and this could cause (if the speech is the target signal <NUM>) that the short-term activity information <NUM> is different between the samples and/or slots carrying the words and the sample and/or slot carrying the pause between the words. In some cases, this can be unacceptable, since this could cause the modification of the remixing parameters between the time instants and/or time slots carrying the word and the time instant and/or time slots carrying the pause between the words. Said in other terms, even if we may want that the speech has a gain which is relatively higher than the gains gained for the background, it is possible that we do not want to modify it instantaneously, since an instantaneous modification is understood as unpleasant by a human listening.

However, it has been understood that, by making use of context information (e.g., 370and/or <NUM>), it is possible to address at least some of these inconveniences. A context based integration block <NUM> is provided.

Block <NUM> may permit to perform an utterance integration (see also section <NUM> below). Block <NUM> may in some examples be described as follows: a cumulative sum of the target signal <NUM> (or one of its processed version <NUM>, <NUM>) and a cumulative sum of first signal <NUM> (<NUM> or <NUM>, or one of its processed versions <NUM> or <NUM>) may be obtained depending on whether activity is detected (based on the activity information <NUM>) for time instants or time slots for which activity is detected. In some examples, all the time instants or all the time slots of an interval in of time instants associated to the same criterion are assigned the same value (e.g. the average of the cumulative sum), and they may be assigned to have the same value. Notably, in case some scattered time instants or time slots are associated to a different criterion (e.g., to the dominant criterion), they may be reassigned to the dominant criterion. In addition or in alternative, the block <NUM> may wait up to a minimum threshold of consecutive time instants or time slots associated to the non-dominant criterion before giving the same value for all the preceding time instants and time slots. This may therefore be an averaging which makes use of temporal context information from the future and/or from the past. Further information is provided in section <NUM>.

The output of the block <NUM> may be an averaged version of the target signal <NUM> (<NUM>) and the first signal <NUM> (<NUM>, <NUM>). A gain computation block <NUM> may be provided. The gain computation block <NUM> may operate according to a constraint (such as a target clearance in the example of the attenuation as shown in <FIG>) <NUM> (e.g. C). The output <NUM> of the gain computation block <NUM> may be a rough gain <NUM>. Reference can also be made to section <NUM> below and an example is provided in formula (<NUM>). A target activity refinement (TAD) block <NUM> may substantially perform a similar operation of the TAD block <NUM> and may provide an activity information <NUM> which may be substantially similar to the short term activity detection <NUM>, but which takes into account a more stable processed version of the signals <NUM>, <NUM> and/or <NUM> (<NUM>). This may be due the fact that the utterance integration permits to tolerate long intervals without activity of the target signal <NUM>. Basically, the gate signal <NUM> as outputted by the TAD refinement block <NUM> provides an activity information of the target signal <NUM>. To give an example taken from <FIG>, the activity information may be "active" in interval <NUM>, without distinctions between the status activity information in the interval <NUM> in the other intervals between tB and tE. (To the contrary, the short-term TAD block <NUM> provides an activity information which is "active" when the speech <NUM> is at a level Lv, while the other intervals, including interval <NUM>, would have given a "non-active" output).

It is noted that the gain computation block <NUM>, as such, defines a remixing criterion which only takes into account the metrics <NUM> (absolute metrics) and/or <NUM> (relative metrics) of the current time instant <NUM>, but does not take into account future or past time instants or slots <NUM> and their metrics <NUM> (relative metrics) and/or <NUM> (absolute metrics). The output <NUM> of the gain computation block <NUM> may therefore be, in some examples, an output which does not provide a variable remixing gain (e.g. it is not smoothed). It is possible to understand the output gain <NUM> as a rough remixing gain which has to be subsequently refined by taking into account metrics (e.g. relative metrics <NUM> and/or absolute metrics <NUM>) on future time instants and/or past time instants. Notably, the gain computation block <NUM> basically embodies the second remixing criterion which is verified, for example, in the third, low gain region of <FIG> (between tB and the end of the interval TOR).

The TAD refinement block <NUM> may be seen as identifying the time intervals in which the second remixing criterion is not to be used. This can be, for example, the high gain region 200H1 of <FIG>, in which, e.g. based on the absolute relative metrics <NUM> and <NUM>, no activity of the target signal <NUM> is detected. It is noted that the inputs <NUM> and <NUM> of the TAD refinement block <NUM> are not necessarily the same of the inputs <NUM> and <NUM> of the short-term TAD block <NUM>, but in some examples at least one (or both) the inputs <NUM> and <NUM> of the TAD refinement block <NUM> may be the same of respectively one of the inputs <NUM> and <NUM> of the short-term TAD block <NUM>.

The activity information <NUM> operates like a gate in gain gating block <NUM>. The activity information <NUM> may discriminate between choosing the first remixing criterion and the second remixing criterion. Notably, the output <NUM> of the block <NUM> (gated gain) is still a rough gain. In the example of <FIG>, the rough gated gain <NUM> (e.g. <NUM>) can take two values:.

With reference to formula (<NUM>) (see below), it may be that the rough gain g(t) is defined as: <MAT>.

In this case, the rough gain (gated gain) <NUM> (<NUM>) may be g(t). The determination between the different gains may be made by taking into account the absolute gate <NUM>, which may be the value G with which the intensity Îs (absolute metrics) is compared, so as to obtain the activity information (which e.g. provides information whether the speech is active). The determination between the different gains may be made by taking into account the target clearance so that, if SNRin(t) > C, then the first criterion (e.g. g(t) = <NUM>) is chosen, otherwise the second criterion (e.g. g(t) = <MAT>). Different ways of defining the rough gains (and/or of determining which remixing criterion each time instant pertain) may be implemented.

It is to be noted that, in examples, the elements <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> shown in <FIG> may be optional. When it is referred to the input signal <NUM> (e.g. input mix) and/or residual signal <NUM> (or more in general first signal <NUM>), it is also possible to refer to their processed version(s), e.g. <NUM> and/or <NUM>. On the other side, when it is referred to the target signal <NUM>, it is also possible to refer to its processed version(s), e.g. <NUM> and/or <NUM>. The signals (or processed version thereof) may be used to obtain, for example, the relative metrics (e.g.,, SNRi) and/or the absolute metrics (e.g., intensities).

At block <NUM> (smoothing block), it is possible to smoothly modify the remixing gain for the noise <NUM> (e.g. actuating a deviation from the remixing criteria). Here, the ramp <NUM>, for example, may be generated. In the time instants and/or time slots in the intermediate region (interval tos), neither the first remixing criterion nor the second remixing criterion is used. To the contrary, temporal context information (as explained above) permits to take into account past time instant(s) and/or future time instant(s). Therefore, the gain can be gradually reduced in the ramp <NUM>. The same would apply in the interval tDR, where an ascending ramp <NUM> is obtained analogously. Reference can also be made to sections <NUM> and <NUM> below. It is also noted that at least one remixing gain <NUM> may be seen as being obtained by refining a rough remixing gain <NUM> or <NUM> by adding an additive component (modifying component) which corrects the rough remixing gain <NUM> or <NUM>, smoothening the obtained remixing gain <NUM>.

In some examples, also the start of the ramp <NUM> or <NUM> (at time instant tA or tR) is based on the knowledge of the future temporal context information: knowing that there will be a change in remixing criterion soon (e.g. within a temporal window <NUM> or <NUM> immediately subsequent to the determined time instant or time slot <NUM>), the deviation may start.

Reference can be made, for example, to formulas (<NUM>) and (<NUM>). Hence, the modifying (correcting) can be based on the immediately preceding and/or subsequent time gain <NUM> (gsmooth(t-<NUM>)) as previously provided. By taking into account the gain as output for the immediately preceding time instant or time slot it is possible to obtain a gradual descending or ascending effect for the gain. This is shown in formula (<NUM>) below is for the descending gains (e.g., ramp <NUM> in the intermediate region in the interval tDS) and formula (<NUM>) is for ramp <NUM> in the interval tDR. It may be stated, therefore, that the rough gain <NUM>, <NUM> is refined by taking into account future and/or past time instants or time slots <NUM>. Block <NUM> may have inputs <NUM> (associated to τatt(t); <NUM> (τrel(t)) and tholdahead (<NUM>) as explained in sections <NUM> and <NUM>. It is noted that τatt is greater than τrel.

As explained above, the control block <NUM> may provide temporal information <NUM> on the current time instant or time slot which will be subsequently used as temporal context information <NUM> (e.g., for subsequent time instants or time slots, and/or for refining a previously obtained rough gain <NUM>, so as to deviate from the rough gain <NUM> to obtain the remixing gain <NUM>). As it will be shown later, the temporal information <NUM> on the current time instant or time slot may include at least one of the output of the utterance integration block <NUM> (e.g., <NUM>, <NUM>) or information associated thereto; rough gain <NUM> and/or activity information (e.g. gate information) <NUM>; a gated gain (e.g., rough gain <NUM>); and/or the at least one remixing gain <NUM> (e.g., gsmooth(t-<NUM>)). Some of these information will be explained in greater detail below.

As explained above, we propose, inter alia, to generate the output mix y(t) (output signal <NUM>) by remixing the estimated sources with a time-varying linear combination: <MAT> where t is the time-index (time slot or time instant) and h(t) and g(t) are the signal-adaptive remixing gains to be determined by the control module (control block <NUM>), for which additional details are given in Sec. This is equivalent to combining ŝ(t) and x(t), i.e., y(t) = k(t)ŝ(t) + z(t)x(t), where k(t) and z(t) are the remixing gains in this case. The remixing gains h(t), g(t), k(t), z(t) can be frequency-dependent or broadband (equal for all frequencies). The following discussion uses broadband gains for illustrating the operations.

On the output mix y(t) (<NUM>) further post-processing can be applied, such as loudness normalization, dynamic range compression, or applying equalization.

The signals are here discussed as they were real-valued time signals, but the same problem could be formulated in the time-frequency domain, e.g., Short-time Fourier Transform (STFT) domain.

The remixing gains <NUM> are in general computed based on features (metrics) of the input signals ŝ(t) (<NUM>) and x(t) <NUM> (and/or potentially of b̂(t) <NUM>) along with a criterion, and parameters that define the desired features of the output mixture y(t). These parameters can be user-defined or fixed by one or more presets.

A prominent feature (metrics) is (or is associated to) the intensity of the signals (which may be the absolute metrics <NUM> and <NUM>). Different ways of quantifying the intensity of a signal can be used here, with different computational requirements. These are for example:.

Another important feature (metrics) is the intensity difference between signals (relative metrics <NUM> and/or <NUM>). Different ways of quantifying the intensity difference exist and are applicable for the proposed method. These may be for example:.

From our experience, it is particularly useful to set condition on the minimum intensity difference (also referred as clearance), leaving the input mix <NUM> unchanged if the estimated intensity difference in it is already big enough.

As an example for the control criterion for computing the remixing gains, let us set a specific value C (target clearance <NUM> in <FIG>) as the desired minimum output SNR (e.g. C corresponds to a high SNR so that the target speech <NUM> is clear and intelligible; see also reference numeral <NUM> in <FIG>), e.g. together with the additional condition (which may be optional) that the input mixture <NUM> shall not be modified when the power of ŝ(t) (target signal <NUM>) is below a certain threshold G (e.g., preventing modification to the original mixture in passages where the target speech is not active).

Considering Eq. (<NUM>) (formula (<NUM>)), the output SNR between the target source signal and the residual signal in the output mixture after applying the remixing gains can be estimated as: <MAT> where w(·) is an optional frequency weighting, e.g., k-weighting [<NUM>]. For the sake of clarity, we can set h(t) = <NUM> and ignore w(·): <MAT> from which it is clear that SNRout(t) can be controlled by g(t).

Our example control criteria require to find g(t) such that SNRout(t) > C (clearance condition), together with the condition that that g(t) = <NUM> if the intensity of ŝ(t) is below a certain threshold G, i.e., Îs(t) < G (gating condition or intensity condition). A time-varying, signal-adaptive, broadband solution can be: <MAT>.

(The solution according to formula (<NUM>) is substantially a solution which takes into account, for each time instant <NUM>, only metrics <NUM> and <NUM> on values of that time instant, without taking into account different (future or past) values. The gating condition and/or the clearance condition may form or be comprised in the criterion condition).

The input SNR (also indicated with SNRi or SNRin) can be estimated as: <MAT>.

If the temporal context would be ignored, the intensities Ix and Îs could be computed as Ix = w(x(t)<NUM>) and Îs = w(ŝ(t)<NUM>), however the temporal context <NUM> can be essential for the esthetical pleasantness of the final result. We may use the temporal context as detailed in Sec. In fact, limitation, time integration, and smoothing may be applied on the remixing gains g(t) and/or on the involved signals (e.g., Îs and SNRin(t) also indicated with SNRi(t)) so to avoid abrupt transitions and pumping, and to generally obtain a smooth and esthetically pleasing output mix.

The smooth gains generated by taking into account the temporal context <NUM> and the final esthetical pleasantness could be referred to as gsmooth(t): <MAT>.

It is possible that gsmooth(t) do not strictly fulfill the criterion used for computing the first gains (rough gain) g(t), e.g., by not fulfilling the instantaneous SNR criterion (e.g. criterion condition) at locations in which large gain changes are smoothed over time (e.g. the above discussed second, intermediate region in <FIG>, i.e. in the interval tDS and/or IDR). However, our experience indicates that despite this, the temporally smoothed gains are preferred by the listeners of the resulting mix. Instead of SNRout(t) > C and Îs (which would be a criterion which does not take into account metrics <NUM>, <NUM> on future or past time instants or time slots <NUM>), estimates of the perceived momentary or short-term loudness (e.g.,[<NUM>]) can be used as intensity measures for the control criteria. Preferences for loudness differences are investigated in [<NUM>, <NUM>]. Other criteria can be based on a partial loudness model [<NUM>] or on time-dependent intelligibility or quality metrics, similarly to [<NUM>]. Also a voice activity detection could be usefully integrated, e.g., by replacing the gating condition Îs < G with a condition based on speech presence probability.

Finally, it is possible to extend the solution of Eq. (<NUM>) to provide gains that are not only time-varying and signal-adaptive, but also frequency-varying.

Also, the control module could take b̂(t) instead of x(t) as input and similar results could be achieved. In other words, in addition to ŝ(t), only one signal between x(t) and b̂(t) is needed for the Control module. Our preference is having access to x(t) (as in <FIG>) instead of b̂(t), in particular if ŝ(t) + b̂(t) ≠ x(t). This preference is motivated by the fact that x(t) could be used, e.g., as quality reference (as mentioned in Sec.

<FIG> illustrates main operations using the temporal context for producing gsmooth(t) (also referred to with <NUM>).

Control module in detail: Operational block diagram of an example of the usage of temporal context for producing gsmooth(t).

A non-essential part of the proposed method contains the automatic adjustment of one or more of the operational parameters of the method, e.g., "Target clearance", "Attack", or "Release". This can be based on the classification of the non-speech parts of the input mix x(t), e.g., if these are dominated by music content or by ambient noise and effects. This information can be used to adjust the "Target clearance <NUM>" accordingly, e.g., to a different value as suggested by the findings in [<NUM>, <NUM>].

Another option is to adjust the remixing parameters based on a quality estimate of the separation. Such an estimate can be done based on ŝ(t) (<NUM>) and x(t) (<NUM>), as presented in [<NUM>] or based on deep neural networks (DNNs), similarly to [<NUM>]. , if the separation quality is low (e.g., because of challenging input mix <NUM>), the smoothing parameters can be set to be more conservative and a smaller clearance can be selected.

The Content classification and Parameter adjustment functionalities are not required for the basic operation of the proposed method, but the parameters can be adjusted also manually or fixed by constant presets. However, a classifier <NUM> may classify a content of the signals <NUM> and/or <NUM> and/or <NUM>. For this purpose, the classifier <NUM> may have a class determiner <NUM> which, for example, distinguishes a first class from a second class, for example speech from non-speech, music or other tonal noises from transient events, whereby both a class of the noises and a number of differentiated classes can be arbitrary. The class determiner <NUM> may provide the determined class to a parameter adjuster <NUM> by means of a class determination signal <NUM>. The classifier <NUM> may be configured to set at least one parameter of the combining and / or the signal attenuation based on a result of the classification. The parameters set by means of the parameter adjuster <NUM> can thus relate to any further operation of the device <NUM>.

In a first stage, the temporal context <NUM> may be used for smoothing the intensities of the inputs ŝ(t) (<NUM>) and x(t) (<NUM>, or more in general the first signal <NUM>). Let us consider the input intensity of x(t) (same operations hold for ŝ(t)). As already mentioned, one way to quantify the intensity of x(t) is to compute the power of the signal filtered so to mimic the frequency response of the human ear: Ix(t) = w(∥x(t)∥<NUM>). This is smoothed, e.g., with a first-order infinite impulse response, IIR, filter: <MAT> where α is a feedback coefficient, e.g., computed from a smoothing time-constant. The smoothed estimate <NUM>, <NUM> can be further transformed into a logarithmic domain to better reflect the magnitude response of the human auditory system. This is referred to as Ex(t) for the input signal <NUM> (or more in general the first signal <NUM>) and as Ês(t) for the target source signal <NUM>.

The smoothed intensity estimates are used for a simple level-based activity detection. A gate signal <NUM> is produced, signaling if Ês(t) is big enough in absolute terms, i.e., it is bigger than an absolute threshold and in relative terms, i.e., compared to Ex(t) with a relative threshold.

More in general, the gate signal <NUM> may represent a short-term activity detection, which indicates the activity of the target signal <NUM> but which may be modified by taking into account the temporal context, for example.

The parameters <NUM> and <NUM> may be an absolute threshold (e.g. so-called "absolute gate" , and also indicated with G) and/or a relative threshold (e.g. so-called "relative gate", which is optional).

If the target source <NUM> is speech, it has to be observed that people tend to talk louder during the first syllables of an utterance. This means that Ês(t) is higher in the utterance beginning compared to the rest of the utterance. Assuming a constant level or the background sources, the effect on the gain is that in the beginning of the utterance less background attenuation is needed than later on and the attenuation changes gradually over time to more attenuation. This "creeping" background attenuation is perceived esthetically rather unpleasing.

UI (e.g. at block <NUM>) takes as the input the TAD output gate signal <NUM> and the two initial signal level estimates Ês(t) and Ex(t) (<NUM> and <NUM>).

UI implements a sliding window mean computation applied on the linear-domain level estimates before transforming them back in the logarithmic domain. The computation has two main modes of operation: start of utterance and sliding. The more interesting is the first one:.

A benefit of this processing is that the level estimate remains constant during the start of an utterance and also later on it changes more slowly. The constant level estimate results into a more consistent gain value and avoid the "creeping gain" problem, making the output esthetically much more pleasant. The output of UI may be refined level estimate Ês(t) and Êb(t). The later may be used, for example, to obtain at least one of the metrics <NUM>, <NUM>, <NUM>, <NUM>.

The window is also called "filtering window" and may make use of values of any of the signals <NUM>, <NUM> (<NUM>, <NUM>) or their processed versions (<NUM>, <NUM>) to obtain filtered versions <NUM> and <NUM> of those signals (<NUM> is the filtered version of <NUM> or <NUM>; <NUM> is the filtered version of <NUM>, e.g. <NUM>, <NUM>, or the processed version <NUM>. A filtering window for the determined current time instant or time slot <NUM> could be, for example, represented by the union of the pluralities of future and past samples <NUM> and <NUM>.

Since the intensity estimates are now temporally more stable, it is beneficial to refine the TAD processing, similarly to Sec. A long-term activity detection <NUM> (here considered a gate signal, e.g. a binary signal) is therefore obtained.

The parameters <NUM> and <NUM> may be an absolute threshold (e.g. G, "absolute gate", which may be the G of formula (<NUM>)) and/or a relative threshold (relative gate, optional).

The core of the gain computation can be now carried out as explained in Sec. <NUM> (see in particular Eq. <NUM>) and by using the stable and smooth intensity estimates and the gate signal obtained so far. The output is g(t), which undergoes a temporal smoothing as explained in the following.

The temporal smoothing can be implemented in various ways, but we may use a simple first-order IIR-filtering approach as an example (other techniques may be implemented). The control inputs to the smoothing method are attack time (<NUM>) tatt (e.g. corresponding to the ramp <NUM> and to the transition from the first remixing criterion to the second remixing criterion), release time (<NUM>) trel (e.g. corresponding to the ramp <NUM> and to the transition from the second remixing criterion to the first remixing criterion), and hold look-ahead time tholdahead (<NUM>). The first two time constants define feedback coefficient values through <MAT> for the attack, and similarly for the release. Other translation formulas may also be used and these are only exemplary. The basic attack/release smoothing produces the smoothed gains: <MAT> where <MAT>.

A problem with this smoothing is that if there is a short pause in the target source signal <NUM>, e.g., between words, sentences, or talkers, the attenuation gain starts the release phase, the background signal comes (partly) back up before being attenuated again when the speech continues. An attempt to solve this pumping problem in the earlier works is to use a constant hold time which delays the release phase always with a constant amount. A drawback of this is that the release is delayed always, regardless if the need for background attenuation continues or not. This can cause unpleasant gaps after the target activity (i.e., speech) has ended. We propose a signal-adaptive mechanism of hold look-ahead for solving this problem: the smoothing uses a look-ahead buffer into the future and detects if the gain applies the same amount or more attenuation within the window of length tholdahead. If this is the case, operation similar to normal hold is activated and the current gain value is kept, otherwise attack and release smoothing is performed normally. This process can be exemplified by surrounding Eq.<NUM> (formula (<NUM>)) with some additional logic: <MAT> where chold(t) is a variable indicating the length of the still remaining time to keep the current gain value and it can be <MAT> otherwise <MAT> where kmin(t) indicates the location of the minimum gain value within a window of tholdahead future values if this value is smaller than the current smoothed value, e.g., <MAT> otherwise <MAT>.

See also <FIG>. Alternative techniques may be implemented.

The description so far is sample-synchronized in the sense that the potential background attenuation induced by applying the produced gains would start exactly at the same sample as the target becomes active. When this is combined with the attack/release-smoothing, the result is that the background attenuation may be perceived to start in a delayed fashion, i.e., too late. Additionally sometimes an earlier attack start of the attenuation is desired for esthetical reasons. A solution is to implement a temporal shift between the gain and the audio signals by shifting the gains by some small time, look-ahead or shift. This operation may conclude the generation of gsmooth(t).

In the case of shifting being used, <FIG> shows the evolution of the at least one gain <NUM> and of the background signal <NUM> after having applied shifting <NUM> (e.g. at the end of method <NUM> and/or <NUM>). In the case of the descending ramp <NUM>, the shifting may move the background signal <NUM> towards the past, e.g. by a first shifting amount (which in this case could be tOA). In the case of the ascending ramp <NUM>, the shifting may move the background signal <NUM> towards the past, e.g. by a second shifting amount. The first shifting amount, for shifting from the first criterion to the second criterion (e.g. when attenuating the background noise), may be different from (e.g. shorter than) the second shifting amount, for shifting from the second criterion to the first criterion (e.g. when the speech ends), but in some other examples the shifting amount may be the same for all the time instants, and a coherent shifting may be applied to all the time instants. In this latter case, it is simply possible to assign an obtained gain gsmooth(t) to a time instant in the past t-Sh (where Sh is a constant number of time instants or time slots, e.g. Sh=<NUM> or another number e.g. between <NUM> and <NUM>), and therefore it is obtained (e.g. at post processing) that the remixing gain provided to the remixing block <NUM> is gsmooth(t-Sh), basically operating a coherent translation towards the past of the obtained at least one gain. In the examples in which there is a different shifting amount between when deviating from the first criterion towards the second criterion and when deviating from the second criterion towards the first criterion, the different shifting amounts may be predefined, e.g., stored in a storage unit: the first shifting amount (e.g. Sh1) will be applied when the transition is from the first criterion towards the second criterion, and the second shifting amount (e.g. Sh2) will be applied when the transition is from the second criterion towards the first criterion. More in general, when shifting is performed, the remixing criteria and the rough gains may be understood as also being shifted towards the past for the same shifting amounts. When shifting is performed, the determined current time instant or time slot may also have the temporal context information <NUM>, which is in the past or in the future with respect to the determined current time instant or time slot before shifting. Subsequently, the obtained gain <NUM> (gsmooth(t)) may be shifted towards the past by the shifting amount (e.g. Sh, Sh1, Sh2).

In addition or alternatively, it is possible to start the ramp directly based on the temporal context information (e.g., by knowing that in the future there is a change of criterion, it is possible to start the deviation).

A (possibly incomplete) list of related works is reported in the following, pointing out commonalities and differences with the approach proposed in this report.

Some advantageous aspect of the present examples are here below briefly resumed:.

Present examples mainly refer to a system (e.g. <NUM>) for processing audio signals. The system (e.g. <NUM>) may comprise a source separation block (e.g. <NUM>) estimating, from an input signal (e.g. <NUM>) which evolves in time along a discrete succession of time instants or time slots (e.g. <NUM>, <NUM>), a target signal (e.g. <NUM>) and at least one residual signal (e.g. <NUM>) to be subsequently remixed (e.g. at remixing block <NUM>, which is part or not part of the system <NUM>) according to at least one remixing gain (e.g. <NUM>) variable along the discrete succession.

The system <NUM> may comprise a control block (e.g. <NUM>) determining, for a determined current time instant or time slot (e.g. <NUM>), at least one metrics (e.g. one of an absolute metrics <NUM> and a relative metrics <NUM>) on the target signal (e.g. <NUM>, <NUM>), or a processed version (e.g. <NUM>, <NUM>) of the target signal (e.g. <NUM>, <NUM>), in the determined current time instant or time slot (e.g. <NUM>). The at least one metrics (e.g. one of an absolute metrics <NUM> and a relative metrics <NUM>) may e.g., be, or be based on at least one relative metrics (e.g. <NUM>) between the target signal (e.g. <NUM>, <NUM>), or a processed version (e.g. <NUM>, <NUM>) of the target signal (e.g. <NUM>, <NUM>), and the input signal (<NUM>, <NUM>), or a processed version (e.g. <NUM>, <NUM>) of the input signal, or the at least one residual signal (<NUM>, <NUM>), or a processed version (<NUM>, <NUM>) thereof, in the determined current time instant or time slot (<NUM>). The at least one metrics may be a relative metrics (e.g. <NUM>). For example the at least one relative metrics may be, or be based on, the SNRin (e.g. signal-to-noise ratio) of the input signal (e.g. <NUM>) or of the processed version thereof (e.g. <NUM> and/or <NUM>). The SNRin (e.g. signal-to-noise ratio) may be, or be associated to, a relative intensity between the target signal (e.g. <NUM>) and the input signal (e.g. <NUM>, <NUM>), or a processed version (e.g. <NUM>, <NUM>) of the input signal, or the at least one residual signal (e.g. <NUM>, <NUM>), or a processed version (e.g. <NUM>, <NUM>) of the at least one residual signal. Examples are provided in formulas (<NUM>) and (<NUM>). For example, according to formula (<NUM>) (see also above) <MAT>, where the Ix and Îs are intensities (or weighted versions of intensities) of the input signal <NUM> (or or a processed version (e.g. <NUM>, <NUM>) of the input signal) and of the target signal <NUM> (or processed version thereof). In some examples, numerals <NUM> and <NUM> of <FIG> are intensities.

The system <NUM> may comprise a temporal context block (e.g. <NUM>). The temporal context block (e.g. <NUM>) may, for example, perform at least one of the operations:.

Therefore, at least one future time instant and at least one past time instant (or at least one of them) may be determined at the temporal context block (e.g. <NUM>). The at least one future time instant or time slot (e.g. <NUM>, <NUM>, <NUM> or one in a window, such as a window <NUM>, <NUM>, <NUM>, <NUM>, etc.) may be, in the discrete succession, after the determined current time instant or time slot (e.g. <NUM>). The past time instant or time slot (e.g. <NUM>, or in a window <NUM>, <NUM>) may be, in the discrete succession, before the determined current time instant or time slot. The temporal context information (e.g. <NUM>) may, for example, be or be based on or at least include at least one metrics on the target signal in at least one future time instant and/or at least one past time instant (e.g. at least one relative metrics <NUM>, at least one absolute metrics <NUM>, or both). The temporal context information (e.g. <NUM>) may, for example, be or be based on or at least include a previously obtained remixing gain (in some examples it may be at least one previously obtained rough remixing gain g(t) e.g. in the future time instants; in some examples it may be at least one previously obtained smoothed, final remixing gain gsmooth(t), and in some other examples it may comprise both, or be, at least one previously obtained rough remixing gain, e.g. in the future time instants, and at least one previously obtained smoothed, final remixing gain gsmooth(t-<NUM>), e.g. for a preceding time instant or time slot).

The control block (e.g. <NUM>) may be configured to generate at least one remixing gain associated to the determined current time instant or time slot by (e.g. <NUM>, t) considering:.

The at least one remixing gain may for example be obtained after having compared the relative metrics (e.g. SNRiin) with a threshold (e.g. C, <NUM>). In some examples (e.g. in the example of formula (<NUM>)), if the relative metrics (e.g. <NUM>) is below the threshold (e.g. SNRiin < C), then there is defined a gain g(t) (e.g. rough gain) such that the distance between the level of the target signal (or processed version thereof) and the level of the level of the input signal (e.g. <NUM>, <NUM>), or a processed version (e.g. <NUM>, <NUM>) of the input signal, or of the at least one residual signal (e.g. <NUM>, <NUM>), or a processed version (e.g. <NUM>, <NUM>) of the at least one residual signal, is increased (e.g. up to C or at least C, e.g. reaching the target clearance <NUM>), e.g. by attenuating the at least one residual signal (e.g. <NUM>), or processed version of the at least one residual signal, and/or by boosting the target signal, or the processed version thereof. If the relative metrics (e.g. SNRiin) is over the threshold (e.g. SNRin > C), then the rough remixing gain g(t) may be maintained as the input gain (e.g. g(t) = <NUM>), since the minimum distance C (e.g. target clearance) is already obtained. Once the rough gain g(t) is obtained, it is possible to modify it by taking into account the temporal context information <NUM>. For example, a smoothed version of the remixing gain gsmooth(t) may be obtained when, from the temporal context information <NUM>, variations of the (e.g. rough) gain in subsequent time instants are determined. For example, if the subsequent time instants are all (or at least prevalently) associated to a different (e.g. rough) gain (e.g. to the attenuating gain), then the rough gain may be modified so as to slightly fade towards the different gain.

In some examples, there are defined at least one first remixing criterion (e.g. implying g(t) = <NUM>) and one second remixing criterion (e.g. implying <MAT> or in any case implying a g(t) which is less than the g(t) at the first remixing criterion) for generating the rough remixing gain (e.g. at the particular determined current time instant t). At least one criterion condition (e.g. a comparison between a relative metrics, e.g. SNRin(t), and a predetermined threshold, e.g. C) may therefore be defined to perform a discrimination between using the first remixing criterion and using the second remixing criterion at each time instant or time slot. In some examples there may be, in addition or alternative, also a comparison between an absolute metrics, such as an intensity Îs(t) of the target signal s(t) with another threshold G, so that if Îs(t) < G, then the rough remixing gain is chosen to be unitary g(t) = <NUM>, otherwise <MAT> if Îs(t) > G (e.g. attenuated background); in some examples (like in formula (<NUM>)), both the criterion conditions may form one OR-condition based on both a first condition (comparison of SNRin with C, or another relative metrics) with a first threshold (C, <NUM>) and another second condition (comparison of intensity Îs(t), or another absolute metrics <NUM>, with a second threshold, e. G, e.g. <NUM>). Therefore on the at least one criterion condition, each time instant or time slot is associated to one of the at least one first remixing criterion and second remixing criterion (e.g. for a first time instant t1 it may be that g(t1) = <NUM> and for a second time instant t2 it may be that <MAT>, this being decided through the evaluation of the criterion condition on the relative metrics and/or the absolute metrics). Hence, at least one criterion condition may be a condition on the at least one (relative and/or absolute) metrics on at least the target signal, or a processed version thereof, at the determined current time instant or time slot, or on information obtained from the at least one metrics on the at least the target signal or a processed version thereof. The determined current time instant or time slot is associated to one of the at least one first remixing criterion and one second remixing criterion based on the metrics on the target signal, or a processed version of the target signal, in the determined current time instant or time slot.

The system may also obtain (e.g. determine) the at least one remixing gain (e.g. in smoothed version in some examples, which is also indicated with gsmooth(t)) for the determined current time slot or time instant (t) by considering temporal context information <NUM> so as to deviate, from the at least one rough remixing gain, based on a deviation obtained from the temporal context information <NUM>. In some examples, by being known that the next future time instants or time slots (totally or partially in a subsequent time window) the rough remixing criterion g(t+Δt) will be different, then some deviations may be possible. It is possible to understand that the deviations permit to obtain a graceful transition from a remixing gain implied by a remixing criterion to another remixing gain implied by another remixing criterion. Examples of deviations are proposed in formulas (<NUM>) and (<NUM>). It is possible to correct the rough remixing gain (<NUM>) by an amount associated to a previously obtained remixing gain for a time instant or time slot preceding the determined current time instant or time slot; this means that the already obtained remixing gain. , in one example at the preceding time instant or time slot t-<NUM> a remixing gain gsmooth(t-<NUM>) has been obtained, and at time instant or time slot t the remixing gain gsmooth(t) may be obtained by correcting the at least one rough remixing gain (<NUM>) by an amount associated to a previously obtained at least one remixing gain (e.g. gsmooht(t-<NUM>) for a time instant or time slot (e.g. <NUM>) preceding the determined current time instant or time slot, like in formulas (<NUM>) and (<NUM>). It is possible, in addition or alternative, to correct the at least one rough remixing gain g(t) through a linear combination of the through remixing gain obtained (e.g. through the evaluation of the criterion condition applied to the relative and/or absolute metrics) for the present current time slot or time instant t and the remixing gain (gsmooth(t-<NUM>)) obtained for the preceding time slot or time instant t-<NUM>. Therefore, by taking into account the temporal context information <NUM> (comprising e.g. information such as gsmooth(t-<NUM>), which is information on the past, and/or information such as the rough gain for subsequent time slots or time instants, which is information on the future) it is possible to properly deviate from the remixing criterion defined by evaluating the criterion condition.

In some examples, the deviation from the rough remixing gain (e.g. g(t)) by correcting the at least one rough remixing gain (e.g. g(t)) for a gain amount associated to a previously obtained remixing gain (e.g. gsmooth(t-<NUM>)) for a time instant or time slot (e.g. t-<NUM>) preceding the determined current time instant or time slot (e.g. t) may be subjected to the fulfilment of a deviation condition. The deviation condition may also be based on the temporal context information <NUM>. In this case, the temporal context information <NUM> may include information on rough remixing gains already obtained for time instants or time slots following the determined time instant or time slot (e.g. in a time window from t, or t+<NUM>, to t+tholdahead, or in some examples another window which is not immediately subsequent to the current time instant or time slot t). The deviation condition may be fulfilled e.g. when a predetermined number (e.g., according to examples, the a predetermined number, or the majority, or all) of rough remixing gains already obtained for time instants or time slots (e.g. in the time window from t, or t+<NUM>, to t+tholdahead) following the determined time instant or time slot (e.g. t) are associated to a remixing criterion which is different from the remixing criterion of the time instant or time slot preceding the current determined time instant or time slot (or a time instant or time slot preceding the time instants or time slots in the time window following the determined time instant or time slot, such as one of the two time instants or time slots, like t-<NUM> and t, immediately preceding the time instants or time slots in the time window following the determined time instant or time slot, e.g. one of the current determined time instant or time slot and the time instant or time slot preceding the current determined time instant or time slot), and otherwise the deviation condition is not fulfilled. If the deviation condition is satisfied, then the deviation is carried out. Otherwise the remixing gain (e.g. g(t)) for the determined current time instant or time slot (e.g. t) may be maintained the same of the at least one remixing gain for a time instant or time slot (e.g. t-<NUM>) preceding the determined current time instant or time slot (e.g. t). For example, if only a low number of subsequent time instants (e.g. in the window from t, or t+<NUM>, to t+tholdahead) is assigned to a different remixing criterion, then the deviation is not performed, but if a great number (e.g. all in some examples) of subsequent time instants (e.g. in the window from t, or t+<NUM>, to t+tholdahead) is assigned to a different remixing criterion, then the deviation is performed. Therefore, disturbance may be tolerated.

In some examples, one remixing criterion may be dominant over another criterion. In some examples the remixing criterion according to which the residual signal <NUM> is attenuated (e.g. when <MAT>) may be dominant over the remixing criterion according to which the residual signal <NUM> is not attenuated (e.g. when g(t) = <NUM>). This because it has been understood that this is preferable, e.g. when the target signal <NUM> is speech and the residual signal <NUM> is noise, so as to avoid an abrupt increase of noise e.g. between two words.

It is to be noted that the system <NUM> may have or may not have the remixing block <NUM>, according to the examples. The remixing block <NUM> may simply receive the target signal <NUM>, and residual signal <NUM> (or the input signal <NUM>) together with the remixing gain (e.g. gsmooth(t)), and the remixing block <NUM> will apply the remixing gain (e.g. gsmooth(t)) to the signal (e.g. to the residual signal). However, in some examples the remixing gain (e.g. gsmooth(t)) is not necessarily to be applied to the signal (e.g., input signal <NUM>, target signal <NUM>, or residual signal <NUM>) at the same time t for which it has been obtained. Indeed, the system <NUM> may shift the at least one remixing gain (gsmooth(t)) as obtained for each time instant or time step of the discrete succession of time instants or time slots by a predetermined number of time instants or time steps towards the past. For example, the remixing gain gsmooth(t) may be assigned to gsmooth(t-D), where D is a predetermined number of time slots or time instants. Hence, a better smoothing may be obtained.

Some additional variants and/or additional or alternative aspects and/or examples are discussed here below.

The gain computation block <NUM> provides at least one gain according to a second criterion (e.g., in the low gain region 200H2 in <FIG>). The gate <NUM> may permit to discriminate between the first criterion and the second criterion, the first criterion providing a unitary gain for both the target signal <NUM> and the background signal <NUM>.

While there is no ramp <NUM> and <NUM> obtained, notwithstanding, the utterance integration block <NUM> may permit to maintain the low gain for the background level <NUM>. This is because the utterance integration block <NUM> has the possibility of looking in the future with the temporal context information <NUM> (<NUM>), which provides metrics <NUM> and/or <NUM> regarding future time instants or time slots <NUM> (or more in detail, a window <NUM> or <NUM> of future time instants or time slots). It is also possible to take into consideration past time instants or time slots, such as those in the window <NUM> or <NUM> immediately preceding the determined current time instant or time slot <NUM>. The utterance integration, therefore, permits to maintain the level at the criterion established for the dominant second remixing criteria at the expense of the non-dominant first remixing criterion. A possibility is provided when transitioning from the second criterion to the first criterion. Other examples may also completely avoid the utterance integration.

Another example is provided by avoiding the utterance integration <NUM> and the short term TAD block <NUM>, but maintaining the blocks <NUM>, <NUM>, <NUM>, and <NUM>, for example. Also in this case, it is possible to obtain a soft transitioning between the two remixing criteria. Information from the future (part of the temporal context information) may also indicate the start of the ramp <NUM> and <NUM> at time instants tA and tR.

In some cases, it is possible (e.g., when the information from the future does not provide the time in the instant in which the ramp shall be started) that the gates <NUM> as provided could, for example, be shifted by a predetermined amount towards the past. However, in some examples, this could be post-processing operation down streamed to block <NUM> (but up streamed to the remixing block <NUM>).

Upstream to the remixing block <NUM>, it is possible to encode a bitstream encoding the target signal (<NUM>), or a processed version (<NUM>, <NUM>) thereof, and the at least one residual signal (<NUM>), or a processed version (<NUM>, <NUM>) thereof, or input signal (<NUM>), or a processed version (<NUM>, <NUM>) thereof, and the at least one gain (<NUM>). The bitstream may be stored and/or transmitted (e.g., through electric or wireless transmissions media) and may be subsequently received, read and decoded upstream to the remixing block <NUM>.

Additionally or alternatively upstream to the control block <NUM>, it is possible to encode a bitstream encoding the target signal (<NUM>), or a processed version (<NUM>, <NUM>) thereof, and the at least one residual signal (<NUM>), or a processed version (<NUM>, <NUM>) thereof, or input signal (<NUM>), or a processed version (<NUM>, <NUM>) thereof. The bitstream may be stored and/or transmitted (e.g., through electric or wireless transmissions media) and may be subsequently received, read and decoded upstream to the control block <NUM>.

Basically, any of blocks <NUM>, <NUM>, <NUM>, <NUM>, may be separated from the other ones or may be in the same device of at least one of the other ones.

Here above reference is often made to the at least one remixing gain mostly using examples in which the gain g(t) (in its rough version) or gsmooth(t) (in its corrected, deviated version) is the remixing gain to be applied to the background noise <NUM> (b(t)). Notwithstanding, it is also possible to apply a gain h(t) (in its rough version) or hsmooth(t) to the target signal <NUM> (s(t)). The at least one gain (either in its rough version or in its smoothed version) may also comprise both the remixing gain to be applied to the background noise <NUM> (b(t)) and the gain h(t) (in its rough version) or hsmooth(t) to the target signal <NUM> (s(t)) and may therefore be formed e.g. by a <NUM>-elements vector.

In some examples, we will have that a second ratio (which may be <NUM>/gsmooth(t), e.g. obtained at the second remixing criterion, when the background signal <NUM> is attenuated) between the rough remixing gain associated to the target signal (which may be <NUM>) and the rough remixing gain (which may be gsmooth(t)<<NUM>) associated to the input signal (or processed version thereof) or the target signal (or processed version thereof) may be higher than a first ratio (which may be <NUM>, e.g. obtained at the first remixing criterion, e.g. non-attenuating the background signal) between the rough remixing gain (which may be <NUM>) associated to the target signal and the rough remixing gain (which may be <NUM>) associated to the input signal (or processed version thereof) or the target signal (or processed version thereof). During the transitional periods, the ratio may be moved from the first ratio to the second ratio, or vice versa.

The examples above also refer to a method for processing audio signals, comprising:.

The examples above also refer to a non-transitory storage unit storing instructions which, when executed by a processor, cause the processer to process audio signals, according to:.

Some examples according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, examples of the present invention may be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine-readable carrier.

Other examples comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier. In other words, an examples of the method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further examples of the methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. A further example is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. A further examples comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further examples comprises a computer having installed thereon the computer program for performing one of the methods described herein.

In some examples, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some examples, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein.

Claim 1:
A system (<NUM>) for processing audio signals, comprising:
a source separation block (<NUM>) configured to estimate, from an input signal (<NUM>) evolving in time along a discrete succession of time instants or time slots (<NUM>, <NUM>), a target signal (<NUM>) and at least one residual signal (<NUM>) to be subsequently remixed (<NUM>) according to at least one remixing gain (<NUM>) variable along the discrete succession;
a control block (<NUM>) configured to determine, for a determined current time instant or time slot (<NUM>), a first, relative metrics (<NUM>) on the target signal (<NUM>, <NUM>), in the determined current time instant or time slot (<NUM>), wherein the first, relative metrics compares a level of the target signal (<NUM>, <NUM>) with a level of the at least one residual signal (<NUM>, <NUM>) or the input signal (<NUM>, <NUM>), in the determined current time instant or time slot (<NUM>); and
a temporal context block (<NUM>) configured to determine temporal context information (<NUM>, <NUM>, <NUM>) based on a second, relative metrics (<NUM>) in at least one future and/or past time instant or time slot (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>), the second, relative metrics (<NUM>) comparing a level of the target signal (<NUM>) with a level of the input signal (<NUM>, <NUM>) or the at least one residual signal (<NUM>, <NUM>), in the at least one future and/or past time instant or time slot (<NUM>), the at least one future time instant or time slot (<NUM>, <NUM>, <NUM>) being, in the discrete succession, after the determined current time instant or time slot (<NUM>), and the past time instant or time slot (<NUM>, <NUM>, <NUM>) being, in the discrete succession, before the determined current time instant or time slot,
wherein the control block (<NUM>) is configured to generate at least one remixing gain (<NUM>) associated to the determined current time instant or time slot based on:
the first, relative metrics (<NUM>) in the determined current time instant or time slot (<NUM>); and
the temporal context information (<NUM>, <NUM>, <NUM>).