Repository: kymatio/kymatio

Issue:
scattering3d returns a tuple instead of tensor
Each implementation has very different expectations of its inputs and outputs:
- 1D: Expects inputs of size `(B, 1, T)`, outputs `(B, P, T/2**J)`, where `P` is the number of scattering coefficients. Note that if the second dimension of the input is not `1`, it errors.
- 2D: Expects inputs of size `(B, C, M, N)`, outputs `(B, C, P, M/2**J, N/2**J)`. Note that the number of dimensions changes here between input and output (this may be desirable but is not consistent with 1D). Also, from what I understand, the second dimension here is essentially treated as a batch dimension.
- 3D: Expects inputs of size `(B, M, N, O)`, outputs a _tuple_ with elements of size `(B, ?, ?, ?)`. Not sure what's going on here.

We should think about harmonizing these to some extent.
scattering3d returns a tuple instead of tensor
Each implementation has very different expectations of its inputs and outputs:
- 1D: Expects inputs of size `(B, 1, T)`, outputs `(B, P, T/2**J)`, where `P` is the number of scattering coefficients. Note that if the second dimension of the input is not `1`, it errors.
- 2D: Expects inputs of size `(B, C, M, N)`, outputs `(B, C, P, M/2**J, N/2**J)`. Note that the number of dimensions changes here between input and output (this may be desirable but is not consistent with 1D). Also, from what I understand, the second dimension here is essentially treated as a batch dimension.
- 3D: Expects inputs of size `(B, M, N, O)`, outputs a _tuple_ with elements of size `(B, ?, ?, ?)`. Not sure what's going on here.

We should think about harmonizing these to some extent.


Discussion:
In my opinion it is the output of `1D` which is wrong here.
It should be `(B, 1, P, T/2**J)`.
So that then we can work on stereo signals or multichannel signals by replacing the `1` with a `C`, like in `2D`.

@louity what do you think of the 3d current input-output layout? I'm asking you because you documented some of it
> In my opinion it is the output of `1D` which is wrong here.
> It should be `(B, 1, P, T/2**J)`.
> So that then we can work on stereo signals or multichannel signals by replacing the `1` with a `C`, like in `2D`.

Agreed. The only thing that gives me pause here is that the first two dimensions behave the same (that is, they are both "batch" dimensions). It feels redundant. Why should the scattering transform care about what dimension is "batch" and what is "channel"?

A solution to this would be to make it independent of number of dimensions. In other words, `Scattering1D` would always act on the last dimension (convert it to two dimensions), `Scattering2D` on the last two (convert to three), and `Scattering3D` on the last three (convert to four). All preceding dimensions would be "batch" dimensions.
> Agreed. The only thing that gives me pause here is that the first two dimensions behave the same (that is, they are both "batch" dimensions). It feels redundant. Why should the scattering transform care about what dimension is "batch" and what is "channel"?

The batch size remains the same at every layer of a deep learning architecture, whereas the number of channels may change (if you're learning "upmixing" for instance, i.e. mono-to-stereo conversion)

> A solution to this would be to make it independent of number of dimensions. In other words, Scattering1D would always act on the last dimension (convert it to two dimensions), Scattering2D on the last two (convert to three), and Scattering3D on the last three (convert to four). All preceding dimensions would be "batch" dimensions.

I like this. But it requires a considerable effort in raising abstraction in what consitute spatial variables vs scattering path variables. This is exactly the sort of abstraction that I developed in my dissertation (context-sensitive grammar in which scattering paths are lists of lists) and implemented in scattering.m but it seems like Kymatio has taken the direction of hard-coding a lot of assumptions as to which variables should map to which subscript
I should add that whatever decision we take should be informed by future plans of implementing routines for multivariables scattering transforms, including:
* roto-translation
* time-frequency
* 1-D x 1-D  (images in which the horizontal and vertical dimensions are not physically homogeneous)
* 1-D x 2-D (videos)
> The batch size remains the same at every layer of a deep learning architecture, whereas the number of channels may change (if you're learning "upmixing" for instance, i.e. mono-to-stereo conversion)

Sure, but why should this make the scattering transform give "channels" special treatment? At the end of the day, it's just an indexing dimension that is not operated on.
> 
> > A solution to this would be to make it independent of number of dimensions. In other words, Scattering1D would always act on the last dimension (convert it to two dimensions), Scattering2D on the last two (convert to three), and Scattering3D on the last three (convert to four). All preceding dimensions would be "batch" dimensions.
> 
> I like this. But it requires a considerable effort in raising abstraction in what consitute spatial variables vs scattering path variables. This is exactly the sort of abstraction that I developed in my dissertation (context-sensitive grammar in which scattering paths are lists of lists) and implemented in scattering.m but it seems like Kymatio has taken the direction of hard-coding a lot of assumptions as to which variables should map to which subscript

Does it? The way I understand it, it should just require "wrapping" the `forward()` methods to do a reshape of the input, save the shape of the indexing/batch dimensions, then reshape the output to match that. We would still keep the convention of `(P, T)` and `(P, M, N)`, etc. for the last three, where `P` indexes the path.

I'm not sure I see much "hard-coding" of assumptions here. The meta information can map the path variable (i.e., the second-to-last in 1D, third-to-last in 2D, etc.) to lists of lists or anything else. The only restriction here is that all the coefficients be of the same `(M, N)` size. Now whether that is a good idea or not is up for debate.

> I should add that whatever decision we take should be informed by future plans of implementing routines for multivariables scattering transforms, including:
> 
>     * roto-translation
> 
>     * time-frequency
> 
>     * 1-D x 1-D  (images in which the horizontal and vertical dimensions are not physically homogeneous)
> 
>     * 1-D x 2-D (videos)

For the first two, I don't think this affects the the conventions. Roto-translation takes images (2D signals) as input, so would fall under `(M, N) -> (P, M/2**J, N/2**J)`. Similarly, time-frequency takes 1D signals and would fall under `(T,) -> (P, T/2**J)`. I think the other two would fall under similar conventions with a scattering path index that would index all the different coefficients.

Am I missing something here?
> Sure, but why should this make the scattering transform give "channels" special treatment? At the end of the day, it's just an indexing dimension that is not operated on.

To preserve the property that batch size remains fixed in every tensor of the computation graph
This property is very important in the age of batch normalization
Therefore I think that we need two dimensions that is not operated on: one to index homogeneous inputs (batch), one to index heterogeneous inputs (channels).
Having more than two is a nice to have but not a must have because we can always boil it down to two dimensions by reshaping.


> I'm not sure I see much "hard-coding" of assumptions here. The meta information can map the path variable (i.e., the second-to-last in 1D, third-to-last in 2D, etc.) to lists of lists or anything else. 

OK these are good news. I stand corrected.


> For the first two, I don't think this affects the the conventions.

Well in time-frequency scattering I could make the case that the output should be (at least optionally)
`(P, T/2**J, F)` where F is frequency so that we could run a 2-d convnet in the time-frequency domain further down the line. Granted the same could be said about time scattering to some extent, on the condition that `j2_min` > `j1_max`. Or is that rearrangement of dimensions something that should be considered post-processing, and completely separate from the bulk of computation?


> > Sure, but why should this make the scattering transform give "channels" special treatment? At the end of the day, it's just an indexing dimension that is not operated on.
> 
> To preserve the property that batch size remains fixed in every tensor of the computation graph
> This property is very important in the age of batch normalization
> Therefore I think that we need two dimensions that is not operated on: one to index homogeneous inputs (batch), one to index heterogeneous inputs (channels).

Of course. The batch size needs to remain the same when computing forward passes in a neural network. That being said, the `Scattering*D` object itself is not a neural network layer. It feels a little "above its pay grade" to think about channels, and so on. All we ask of it is to transform some data.

> Having more than two is a nice to have but not a must have because we can always boil it down to two dimensions by reshaping.

It's not just having more than two, but also less than two. I was discussing this with @eickenberg earlier (I forget where). It would be nice just to call a 1D scattering transform on a `(T,)` vector and get a `(P, T/2**J)` vector out. Or if we have a batch with only one channel, to call it on `(B, T)`, and so on. I really think such a dimension-agnostic (i.e. only acts on the last `d` dimensions) would make a for a more predictable interface.

> > I'm not sure I see much "hard-coding" of assumptions here. The meta information can map the path variable (i.e., the second-to-last in 1D, third-to-last in 2D, etc.) to lists of lists or anything else.
> 
> OK these are good news. I stand corrected.

While this information is in there, only 1D has a `meta()` function right now, and it's a bit of a mess. It would be good to think about how to define this in a consistent way for all scattering transforms in Kymatio (and beyond). I don't understand your lists of lists approach completely, but something like that could be good to have.

> > For the first two, I don't think this affects the the conventions.
> 
> Well in time-frequency scattering I could make the case that the output should be (at least optionally)
> `(P, T/2**J, F)` where F is frequency so that we could run a 2-d convnet in the time-frequency domain further down the line. Granted the same could be said about time scattering to some extent, on the condition that `j2_min` > `j1_max`. Or is that rearrangement of dimensions something that should be considered post-processing, and completely separate from the bulk of computation?

Sure, but, as you say, the `T-F` plane would have different sizes depending on `P`. It's not a "rectangular" tensor. What to do with the remaining space, fill with NaNs? I think this could get quite hairy. That being said, there should be easy ways to extract such slices, if desired, for the reasons you mention.
>  That being said, the Scattering*D object itself is not a neural network layer. It feels a little "above its pay grade" to think about channels, and so on. All we ask of it is to transform some data.

I disagree. from what I understand, a big advantage of Kymatio is that Scattering*D can be fed as input or output to a neural network layer. Isn't that part of the spec?
Also note that the batch size may change between training and prediction but the number of channels may not.
So the fundamental difference between batch and channel is that batch remains constant from layer to layer but may vary from pass to pass whereas it's vice versa for channels.


>  I was discussing this with @eickenberg earlier (I forget where). It would be nice just to call a 1D scattering transform on a (T,) vector and get a (P, T/2**J) vector out. Or if we have a batch with only one channel, to call it on (B, T), and so on. I really think such a dimension-agnostic (i.e. only acts on the last d dimensions) would make a for a more predictable interface.

But then how do you make the difference between (B, T) and (C, T) ? Isn't the conflation between batch and channel a potential source of bugs in pytorch autodifferentiation?


> While this information is in there, only 1D has a meta() function right now, and it's a bit of a mess. It wSure, but, as you say, the T-F plane would have different sizes depending on P. It's not a "rectangular" tensor. What to do with the remaining space, fill with NaNs? I think this could get quite hairy.

You are right but it becomes rectangular on the condition that j1_max < j2_min which is not unheard of in audio. No need for NaNs in this case. But we can always fill with zeros in the general case. It preserves energy. I'm OK with leaving this as post-processing but that represents a big operation of memory copy, which will slow down the code considerably. One of the core reasons why scattering.m is so fast is that the IFFT happens exactly where the coefficients need to be placed for the net layer.

> > That being said, the Scattering*D object itself is not a neural network layer. It feels a little "above its pay grade" to think about channels, and so on. All we ask of it is to transform some data.
>
> I disagree. from what I understand, a big advantage of Kymatio is that Scattering*D can be fed as input or output to a neural network layer. Isn't that part of the spec?

I'm not sure. For one, it doesn't inherit from `torch.nn.Module`, which seems to be the standard way to do this.

> Also note that the batch size may change between training and prediction but the number of channels may not.
> So the fundamental difference between batch and channel is that batch remains constant from layer to layer but may vary from pass to pass whereas it's vice versa for channels.

Sure, but to the scattering transform they're all the same (they're all batch-like dimensions). It doesn't do anything special with the channel dimension (except in the 1D case, where it errors if it's not size 1). Why give it special status? I understand that in a network, it may have special status, but why should the scattering transform care about this? `torch.fft` doesn't care.

> > It would be nice just to call a 1D scattering transform on a (T,) vector and get a (P, T/2**J) vector out. Or if we have a batch with only one channel, to call it on (B, T), and so on. I really think such a dimension-agnostic (i.e. only acts on the last d dimensions) would make a for a more predictable interface.
> 
> But then how do you make the difference between (B, T) and (C, T) ? Isn't the conflation between batch and channel a potential source of bugs in pytorch autodifferentiation?

Do you need to know the difference? Specifically, does the scattering transform need to? The computation, and the gradient, is the same. I don't see how it would break down the automatic differentiation. Could some PyTorch folks chime in here? @eickenberg @edouardoyallon?

> > Sure, but, as you say, the T-F plane would have different sizes depending on P. It's not a "rectangular" tensor. What to do with the remaining space, fill with NaNs? I think this could get quite hairy.
> 
> You are right but it becomes rectangular on the condition that j1_max < j2_min which is not unheard of in audio. No need for NaNs in this case. But we can always fill with zeros in the general case. It preserves energy. I'm OK with leaving this as post-processing but that represents a big operation of memory copy, which will slow down the code considerably. One of the core reasons why scattering.m is so fast is that the IFFT happens exactly where the coefficients need to be placed for the net layer.

It's not clear to me what `j1_max < j2_min` implies. At any rate, that's quite a strong constraint to impose to suit a scattering regime that is "not unheard of." Sure, you can fill with zeros and you may improve performance by copying less memory for some tasks, but for other tasks you're decreasing performance by unnecessarily multiplying by lots of zeros and increasing memory usage.
The 3D version returns different outputs for different versions.

- "standard" would return `(B,P, M/2**J,N/2**J,O/2**J)` 
- "integral" would return `(B,P,1,1,1)` (or `(B,P)` not sure)
- "local" would return `(B,P,1)`

The reason for "integral" is practically dimensionality reduction (among others) because  3D  is typically large. For "local" you have to ask @louity to be certain but I think it's to identify interesting regions in the moduli (e.g. midpoints between bonded atoms in quantum chemistry)
Cool. So can `integral` and `local` be computed from `standard`? The more we can standardize the function of the `forward()` functions, the better. It may be better to apply these transformations as post-processing, if possible.

Also, what's the purpose of the tuple structure? Is that necessary or could the Tensors be concatenated?
local probably not. Integral yes but it may be slower.

> Cool. So can `integral` and `local` be computed from `standard`? The more we can standardize the function of the `forward()` functions, the better. It may be better to apply these transformations as post-processing, if possible.
> 
> Also, what's the purpose of the tuple structure? Is that necessary or could the Tensors be concatenated?

Yes I can do that

> local probably not.

Hmm ok. Maybe best to leave this right now. Is it something that's used anywhere else?

> Integral yes but it may be slower.

Slower because it would leave large arrays unaveraged? As long as they stay in GPU memory, is it that bad? The caller can always average them later, right?

> > Also, what's the purpose of the tuple structure? Is that necessary or could the Tensors be concatenated?
> 
> Yes I can do that

Ok. Let's do that, then.
FWIW "local" measures scattering transform near a single point, much in the same way as the foveal scattering of @AndreuxMath and @beedotkiran
So it makes sense that it does not have spatial coordinates
Likewise for integral
@janden we might consider implementing integral in 1d in the future, as it runs faster than convolution with phi

@gexarcha I think that there are two issues here. One is how to interface local and integral with standard. The other is the output type: tuple vs tensor. The former can be postponed to `v0.2`, but the output type issue needs to be resolved before we release `v0.1.0-alpha`.
(Woa I thought we had lost this issue somewhere, but it just has a new title :) )

I think we are probably in the luxurious situation where we can just operate on the last n relevant axes of our data, e.g. by reshaping everything else into one huge sample axis and then reshaping back after the operation is done.

It would accommodate batchnorm and all the other shape-bound issues that might come up, since we reshape back after. 

The only thing that might happen is that the initial reshape triggers a copy. But, most probably, if it did trigger a copy, then the GPU wouldn't  have been able to handle its ordering without making a copy anyway.

This relatively liberal input shape requirement has been the general practice I have done in code that I write, but I have gotten repeated feedback stating that it becomes hard to read (OK maybe at its most extreme it was, since it did 1D,2D,3D all in one). So we should watch out for that if we proceed this way.

In general this has the advantage of flexibility but the disadvantage of maybe doing unexpected stuff.


@eickenberg yes, the reason why I renamed the issue is that it contains several debated decisions, which do not have the same timeline.
1. First there is the question of output type. In my mind, this needs to be resolved before alpha release, because we will necessarily be drawn towards more consistency, not less, so it's best to avoid any API-breaking decision — especially given that a change in output type cannot be nicely deprecated. So I would like us to focus on closing that issue first, while we can afford some time to think about the rest.

2. Then there is the question of kwarg API: "local", "integral", etc. It would be nice to clarify this, but it's not a must-have as it can be deprecated in v0.2, and definitely modified in v0.3

3. Then there is the question of allowing for more generality in the dimensionality of the input. This is a great avenue for improvement, and I will personally devote some time to it in the future; but I don't view it as release blocking, because we can always add more generality further down the line. And if our decision ends up being API-breaking, there's always a possibility to deprecate a certain usage pattern, simply by looking input the `ndims` of `x` (as is done in ScatNet 1d for instance).

4. Lastly there is the question on how Kymatio vectorizes computations under the hood. Because we don't cache our tensors nor perform any in-place operations, this can be changed overnight without the user realizing it, so as you say it is indeed a luxurious situation.

But I just want us to be on the same page on the fact that, although point 4 is certainly luxurious, point 1 is absolutely not luxurious and could become quite thorny if we don't resolve it by 0.1-alpha
I should add that thinking about this issue made me, personally, come to the conclusion that we need not assume that `M`, `N`, `O` must be specified in advance by the user. By default, they could very well be inferred from `J`. This would allow to have exactly the same API for all dimensions, which in my mind would be the pinnacle of consistency and clarity.
Regarding the relationship between ndims of input and ndims of output, it is true that it would be good to be more consistent. At the beginning I wanted to modify the output of `Scattering1D`, but upon second thought I think it's best to modify the input.
So that: `(B,T)` maps to `(B,P,T')`
just like in other modules.
Then in the future we can think about implementing something more generic, as @janden and @eickenberg suggested:
`(T,)` -> `(P,T')`
`(B,T)` -> `(B,P,T')`
`(B,C,T)`->`(B,C,P,T')`
and so forth.
I can open a PR for that.
> I think we are probably in the luxurious situation where we can just operate on the last n relevant axes of our data, e.g. by reshaping everything else into one huge sample axis and then reshaping back after the operation is done.
>
> ...
>
> In general this has the advantage of flexibility but the disadvantage of maybe doing unexpected stuff.

Agreed. I don't think so, as long as we're clear about the behavior. At the end of the day, it's not terribly more "unexpected" than `torch.fft`.

> 1. First there is the question of output type. In my mind, this needs to be resolved before alpha release, because we will necessarily be drawn towards more consistency, not less, so it's best to avoid any API-breaking decision — especially given that a change in output type cannot be nicely deprecated. So I would like us to focus on closing that issue first, while we can afford some time to think about the rest.

Agreed. I think from the above discussing, we'd be fine concatenating everything into one big Tensor (as done in 1D and 2D). The non-standard outputs would work here, as well, with just a `(1, 1, 1)` shape for the spatial coordinate (or without a coordinate at all).

> 2. Then there is the question of kwarg API: "local", "integral", etc. It would be nice to clarify this, but it's not a must-have as it can be deprecated in v0.2, and definitely modified in v0.3

Yes I think that makes sense. I'd also like to get an idea of how common these are. If they're not crucial, we could take them out now and add them back in later once we've decided on the interface. That's probably overkill, though.

> 3. Then there is the question of allowing for more generality in the dimensionality of the input. This is a great avenue for improvement, and I will personally devote some time to it in the future; but I don't view it as release blocking, because we can always add more generality further down the line. And if our decision ends up being API-breaking, there's always a possibility to deprecate a certain usage pattern, simply by looking input the `ndims` of `x` (as is done in ScatNet 1d for instance).

Is this the same thing that @eickenberg was proposing above? On my first read, I saw this as allowing sizes to change at runtime, which sounds like a huge task.

> 4. Lastly there is the question on how Kymatio vectorizes computations under the hood. Because we don't cache our tensors nor perform any in-place operations, this can be changed overnight without the user realizing it, so as you say it is indeed a luxurious situation.

Yes. We want to put ourselves in as flexible a position as possible to allow this down the road.
Also, while we're here, 3D doesn't compute the zeroth order. Can we add this in? Currently, there's another function `compute_integrals` which does this (`test_scattering3d.py` uses this to get the full transform and compare it with the stored data).
yes, 3-D should include the zeroth order.
This can be done in a unified API by defining a list `orders` (defaulting to `range(1+max_order)`) in the `Scattering*D` instantiated object.

Working on the tuple vs tensor inconsistency at the moment.

In my opinion it is the output of `1D` which is wrong here.
It should be `(B, 1, P, T/2**J)`.
So that then we can work on stereo signals or multichannel signals by replacing the `1` with a `C`, like in `2D`.

@louity what do you think of the 3d current input-output layout? I'm asking you because you documented some of it
> In my opinion it is the output of `1D` which is wrong here.
> It should be `(B, 1, P, T/2**J)`.
> So that then we can work on stereo signals or multichannel signals by replacing the `1` with a `C`, like in `2D`.

Agreed. The only thing that gives me pause here is that the first two dimensions behave the same (that is, they are both "batch" dimensions). It feels redundant. Why should the scattering transform care about what dimension is "batch" and what is "channel"?

A solution to this would be to make it independent of number of dimensions. In other words, `Scattering1D` would always act on the last dimension (convert it to two dimensions), `Scattering2D` on the last two (convert to three), and `Scattering3D` on the last three (convert to four). All preceding dimensions would be "batch" dimensions.
> Agreed. The only thing that gives me pause here is that the first two dimensions behave the same (that is, they are both "batch" dimensions). It feels redundant. Why should the scattering transform care about what dimension is "batch" and what is "channel"?

The batch size remains the same at every layer of a deep learning architecture, whereas the number of channels may change (if you're learning "upmixing" for instance, i.e. mono-to-stereo conversion)

> A solution to this would be to make it independent of number of dimensions. In other words, Scattering1D would always act on the last dimension (convert it to two dimensions), Scattering2D on the last two (convert to three), and Scattering3D on the last three (convert to four). All preceding dimensions would be "batch" dimensions.

I like this. But it requires a considerable effort in raising abstraction in what consitute spatial variables vs scattering path variables. This is exactly the sort of abstraction that I developed in my dissertation (context-sensitive grammar in which scattering paths are lists of lists) and implemented in scattering.m but it seems like Kymatio has taken the direction of hard-coding a lot of assumptions as to which variables should map to which subscript
I should add that whatever decision we take should be informed by future plans of implementing routines for multivariables scattering transforms, including:
* roto-translation
* time-frequency
* 1-D x 1-D  (images in which the horizontal and vertical dimensions are not physically homogeneous)
* 1-D x 2-D (videos)
> The batch size remains the same at every layer of a deep learning architecture, whereas the number of channels may change (if you're learning "upmixing" for instance, i.e. mono-to-stereo conversion)

Sure, but why should this make the scattering transform give "channels" special treatment? At the end of the day, it's just an indexing dimension that is not operated on.
> 
> > A solution to this would be to make it independent of number of dimensions. In other words, Scattering1D would always act on the last dimension (convert it to two dimensions), Scattering2D on the last two (convert to three), and Scattering3D on the last three (convert to four). All preceding dimensions would be "batch" dimensions.
> 
> I like this. But it requires a considerable effort in raising abstraction in what consitute spatial variables vs scattering path variables. This is exactly the sort of abstraction that I developed in my dissertation (context-sensitive grammar in which scattering paths are lists of lists) and implemented in scattering.m but it seems like Kymatio has taken the direction of hard-coding a lot of assumptions as to which variables should map to which subscript

Does it? The way I understand it, it should just require "wrapping" the `forward()` methods to do a reshape of the input, save the shape of the indexing/batch dimensions, then reshape the output to match that. We would still keep the convention of `(P, T)` and `(P, M, N)`, etc. for the last three, where `P` indexes the path.

I'm not sure I see much "hard-coding" of assumptions here. The meta information can map the path variable (i.e., the second-to-last in 1D, third-to-last in 2D, etc.) to lists of lists or anything else. The only restriction here is that all the coefficients be of the same `(M, N)` size. Now whether that is a good idea or not is up for debate.

> I should add that whatever decision we take should be informed by future plans of implementing routines for multivariables scattering transforms, including:
> 
>     * roto-translation
> 
>     * time-frequency
> 
>     * 1-D x 1-D  (images in which the horizontal and vertical dimensions are not physically homogeneous)
> 
>     * 1-D x 2-D (videos)

For the first two, I don't think this affects the the conventions. Roto-translation takes images (2D signals) as input, so would fall under `(M, N) -> (P, M/2**J, N/2**J)`. Similarly, time-frequency takes 1D signals and would fall under `(T,) -> (P, T/2**J)`. I think the other two would fall under similar conventions with a scattering path index that would index all the different coefficients.

Am I missing something here?
> Sure, but why should this make the scattering transform give "channels" special treatment? At the end of the day, it's just an indexing dimension that is not operated on.

To preserve the property that batch size remains fixed in every tensor of the computation graph
This property is very important in the age of batch normalization
Therefore I think that we need two dimensions that is not operated on: one to index homogeneous inputs (batch), one to index heterogeneous inputs (channels).
Having more than two is a nice to have but not a must have because we can always boil it down to two dimensions by reshaping.


> I'm not sure I see much "hard-coding" of assumptions here. The meta information can map the path variable (i.e., the second-to-last in 1D, third-to-last in 2D, etc.) to lists of lists or anything else. 

OK these are good news. I stand corrected.


> For the first two, I don't think this affects the the conventions.

Well in time-frequency scattering I could make the case that the output should be (at least optionally)
`(P, T/2**J, F)` where F is frequency so that we could run a 2-d convnet in the time-frequency domain further down the line. Granted the same could be said about time scattering to some extent, on the condition that `j2_min` > `j1_max`. Or is that rearrangement of dimensions something that should be considered post-processing, and completely separate from the bulk of computation?


> > Sure, but why should this make the scattering transform give "channels" special treatment? At the end of the day, it's just an indexing dimension that is not operated on.
> 
> To preserve the property that batch size remains fixed in every tensor of the computation graph
> This property is very important in the age of batch normalization
> Therefore I think that we need two dimensions that is not operated on: one to index homogeneous inputs (batch), one to index heterogeneous inputs (channels).

Of course. The batch size needs to remain the same when computing forward passes in a neural network. That being said, the `Scattering*D` object itself is not a neural network layer. It feels a little "above its pay grade" to think about channels, and so on. All we ask of it is to transform some data.

> Having more than two is a nice to have but not a must have because we can always boil it down to two dimensions by reshaping.

It's not just having more than two, but also less than two. I was discussing this with @eickenberg earlier (I forget where). It would be nice just to call a 1D scattering transform on a `(T,)` vector and get a `(P, T/2**J)` vector out. Or if we have a batch with only one channel, to call it on `(B, T)`, and so on. I really think such a dimension-agnostic (i.e. only acts on the last `d` dimensions) would make a for a more predictable interface.

> > I'm not sure I see much "hard-coding" of assumptions here. The meta information can map the path variable (i.e., the second-to-last in 1D, third-to-last in 2D, etc.) to lists of lists or anything else.
> 
> OK these are good news. I stand corrected.

While this information is in there, only 1D has a `meta()` function right now, and it's a bit of a mess. It would be good to think about how to define this in a consistent way for all scattering transforms in Kymatio (and beyond). I don't understand your lists of lists approach completely, but something like that could be good to have.

> > For the first two, I don't think this affects the the conventions.
> 
> Well in time-frequency scattering I could make the case that the output should be (at least optionally)
> `(P, T/2**J, F)` where F is frequency so that we could run a 2-d convnet in the time-frequency domain further down the line. Granted the same could be said about time scattering to some extent, on the condition that `j2_min` > `j1_max`. Or is that rearrangement of dimensions something that should be considered post-processing, and completely separate from the bulk of computation?

Sure, but, as you say, the `T-F` plane would have different sizes depending on `P`. It's not a "rectangular" tensor. What to do with the remaining space, fill with NaNs? I think this could get quite hairy. That being said, there should be easy ways to extract such slices, if desired, for the reasons you mention.
>  That being said, the Scattering*D object itself is not a neural network layer. It feels a little "above its pay grade" to think about channels, and so on. All we ask of it is to transform some data.

I disagree. from what I understand, a big advantage of Kymatio is that Scattering*D can be fed as input or output to a neural network layer. Isn't that part of the spec?
Also note that the batch size may change between training and prediction but the number of channels may not.
So the fundamental difference between batch and channel is that batch remains constant from layer to layer but may vary from pass to pass whereas it's vice versa for channels.


>  I was discussing this with @eickenberg earlier (I forget where). It would be nice just to call a 1D scattering transform on a (T,) vector and get a (P, T/2**J) vector out. Or if we have a batch with only one channel, to call it on (B, T), and so on. I really think such a dimension-agnostic (i.e. only acts on the last d dimensions) would make a for a more predictable interface.

But then how do you make the difference between (B, T) and (C, T) ? Isn't the conflation between batch and channel a potential source of bugs in pytorch autodifferentiation?


> While this information is in there, only 1D has a meta() function right now, and it's a bit of a mess. It wSure, but, as you say, the T-F plane would have different sizes depending on P. It's not a "rectangular" tensor. What to do with the remaining space, fill with NaNs? I think this could get quite hairy.

You are right but it becomes rectangular on the condition that j1_max < j2_min which is not unheard of in audio. No need for NaNs in this case. But we can always fill with zeros in the general case. It preserves energy. I'm OK with leaving this as post-processing but that represents a big operation of memory copy, which will slow down the code considerably. One of the core reasons why scattering.m is so fast is that the IFFT happens exactly where the coefficients need to be placed for the net layer.

> > That being said, the Scattering*D object itself is not a neural network layer. It feels a little "above its pay grade" to think about channels, and so on. All we ask of it is to transform some data.
>
> I disagree. from what I understand, a big advantage of Kymatio is that Scattering*D can be fed as input or output to a neural network layer. Isn't that part of the spec?

I'm not sure. For one, it doesn't inherit from `torch.nn.Module`, which seems to be the standard way to do this.

> Also note that the batch size may change between training and prediction but the number of channels may not.
> So the fundamental difference between batch and channel is that batch remains constant from layer to layer but may vary from pass to pass whereas it's vice versa for channels.

Sure, but to the scattering transform they're all the same (they're all batch-like dimensions). It doesn't do anything special with the channel dimension (except in the 1D case, where it errors if it's not size 1). Why give it special status? I understand that in a network, it may have special status, but why should the scattering transform care about this? `torch.fft` doesn't care.

> > It would be nice just to call a 1D scattering transform on a (T,) vector and get a (P, T/2**J) vector out. Or if we have a batch with only one channel, to call it on (B, T), and so on. I really think such a dimension-agnostic (i.e. only acts on the last d dimensions) would make a for a more predictable interface.
> 
> But then how do you make the difference between (B, T) and (C, T) ? Isn't the conflation between batch and channel a potential source of bugs in pytorch autodifferentiation?

Do you need to know the difference? Specifically, does the scattering transform need to? The computation, and the gradient, is the same. I don't see how it would break down the automatic differentiation. Could some PyTorch folks chime in here? @eickenberg @edouardoyallon?

> > Sure, but, as you say, the T-F plane would have different sizes depending on P. It's not a "rectangular" tensor. What to do with the remaining space, fill with NaNs? I think this could get quite hairy.
> 
> You are right but it becomes rectangular on the condition that j1_max < j2_min which is not unheard of in audio. No need for NaNs in this case. But we can always fill with zeros in the general case. It preserves energy. I'm OK with leaving this as post-processing but that represents a big operation of memory copy, which will slow down the code considerably. One of the core reasons why scattering.m is so fast is that the IFFT happens exactly where the coefficients need to be placed for the net layer.

It's not clear to me what `j1_max < j2_min` implies. At any rate, that's quite a strong constraint to impose to suit a scattering regime that is "not unheard of." Sure, you can fill with zeros and you may improve performance by copying less memory for some tasks, but for other tasks you're decreasing performance by unnecessarily multiplying by lots of zeros and increasing memory usage.
The 3D version returns different outputs for different versions.

- "standard" would return `(B,P, M/2**J,N/2**J,O/2**J)` 
- "integral" would return `(B,P,1,1,1)` (or `(B,P)` not sure)
- "local" would return `(B,P,1)`

The reason for "integral" is practically dimensionality reduction (among others) because  3D  is typically large. For "local" you have to ask @louity to be certain but I think it's to identify interesting regions in the moduli (e.g. midpoints between bonded atoms in quantum chemistry)
Cool. So can `integral` and `local` be computed from `standard`? The more we can standardize the function of the `forward()` functions, the better. It may be better to apply these transformations as post-processing, if possible.

Also, what's the purpose of the tuple structure? Is that necessary or could the Tensors be concatenated?
local probably not. Integral yes but it may be slower.

> Cool. So can `integral` and `local` be computed from `standard`? The more we can standardize the function of the `forward()` functions, the better. It may be better to apply these transformations as post-processing, if possible.
> 
> Also, what's the purpose of the tuple structure? Is that necessary or could the Tensors be concatenated?

Yes I can do that

> local probably not.

Hmm ok. Maybe best to leave this right now. Is it something that's used anywhere else?

> Integral yes but it may be slower.

Slower because it would leave large arrays unaveraged? As long as they stay in GPU memory, is it that bad? The caller can always average them later, right?

> > Also, what's the purpose of the tuple structure? Is that necessary or could the Tensors be concatenated?
> 
> Yes I can do that

Ok. Let's do that, then.
FWIW "local" measures scattering transform near a single point, much in the same way as the foveal scattering of @AndreuxMath and @beedotkiran
So it makes sense that it does not have spatial coordinates
Likewise for integral
@janden we might consider implementing integral in 1d in the future, as it runs faster than convolution with phi

@gexarcha I think that there are two issues here. One is how to interface local and integral with standard. The other is the output type: tuple vs tensor. The former can be postponed to `v0.2`, but the output type issue needs to be resolved before we release `v0.1.0-alpha`.
(Woa I thought we had lost this issue somewhere, but it just has a new title :) )

I think we are probably in the luxurious situation where we can just operate on the last n relevant axes of our data, e.g. by reshaping everything else into one huge sample axis and then reshaping back after the operation is done.

It would accommodate batchnorm and all the other shape-bound issues that might come up, since we reshape back after. 

The only thing that might happen is that the initial reshape triggers a copy. But, most probably, if it did trigger a copy, then the GPU wouldn't  have been able to handle its ordering without making a copy anyway.

This relatively liberal input shape requirement has been the general practice I have done in code that I write, but I have gotten repeated feedback stating that it becomes hard to read (OK maybe at its most extreme it was, since it did 1D,2D,3D all in one). So we should watch out for that if we proceed this way.

In general this has the advantage of flexibility but the disadvantage of maybe doing unexpected stuff.


@eickenberg yes, the reason why I renamed the issue is that it contains several debated decisions, which do not have the same timeline.
1. First there is the question of output type. In my mind, this needs to be resolved before alpha release, because we will necessarily be drawn towards more consistency, not less, so it's best to avoid any API-breaking decision — especially given that a change in output type cannot be nicely deprecated. So I would like us to focus on closing that issue first, while we can afford some time to think about the rest.

2. Then there is the question of kwarg API: "local", "integral", etc. It would be nice to clarify this, but it's not a must-have as it can be deprecated in v0.2, and definitely modified in v0.3

3. Then there is the question of allowing for more generality in the dimensionality of the input. This is a great avenue for improvement, and I will personally devote some time to it in the future; but I don't view it as release blocking, because we can always add more generality further down the line. And if our decision ends up being API-breaking, there's always a possibility to deprecate a certain usage pattern, simply by looking input the `ndims` of `x` (as is done in ScatNet 1d for instance).

4. Lastly there is the question on how Kymatio vectorizes computations under the hood. Because we don't cache our tensors nor perform any in-place operations, this can be changed overnight without the user realizing it, so as you say it is indeed a luxurious situation.

But I just want us to be on the same page on the fact that, although point 4 is certainly luxurious, point 1 is absolutely not luxurious and could become quite thorny if we don't resolve it by 0.1-alpha
I should add that thinking about this issue made me, personally, come to the conclusion that we need not assume that `M`, `N`, `O` must be specified in advance by the user. By default, they could very well be inferred from `J`. This would allow to have exactly the same API for all dimensions, which in my mind would be the pinnacle of consistency and clarity.
Regarding the relationship between ndims of input and ndims of output, it is true that it would be good to be more consistent. At the beginning I wanted to modify the output of `Scattering1D`, but upon second thought I think it's best to modify the input.
So that: `(B,T)` maps to `(B,P,T')`
just like in other modules.
Then in the future we can think about implementing something more generic, as @janden and @eickenberg suggested:
`(T,)` -> `(P,T')`
`(B,T)` -> `(B,P,T')`
`(B,C,T)`->`(B,C,P,T')`
and so forth.
I can open a PR for that.
> I think we are probably in the luxurious situation where we can just operate on the last n relevant axes of our data, e.g. by reshaping everything else into one huge sample axis and then reshaping back after the operation is done.
>
> ...
>
> In general this has the advantage of flexibility but the disadvantage of maybe doing unexpected stuff.

Agreed. I don't think so, as long as we're clear about the behavior. At the end of the day, it's not terribly more "unexpected" than `torch.fft`.

> 1. First there is the question of output type. In my mind, this needs to be resolved before alpha release, because we will necessarily be drawn towards more consistency, not less, so it's best to avoid any API-breaking decision — especially given that a change in output type cannot be nicely deprecated. So I would like us to focus on closing that issue first, while we can afford some time to think about the rest.

Agreed. I think from the above discussing, we'd be fine concatenating everything into one big Tensor (as done in 1D and 2D). The non-standard outputs would work here, as well, with just a `(1, 1, 1)` shape for the spatial coordinate (or without a coordinate at all).

> 2. Then there is the question of kwarg API: "local", "integral", etc. It would be nice to clarify this, but it's not a must-have as it can be deprecated in v0.2, and definitely modified in v0.3

Yes I think that makes sense. I'd also like to get an idea of how common these are. If they're not crucial, we could take them out now and add them back in later once we've decided on the interface. That's probably overkill, though.

> 3. Then there is the question of allowing for more generality in the dimensionality of the input. This is a great avenue for improvement, and I will personally devote some time to it in the future; but I don't view it as release blocking, because we can always add more generality further down the line. And if our decision ends up being API-breaking, there's always a possibility to deprecate a certain usage pattern, simply by looking input the `ndims` of `x` (as is done in ScatNet 1d for instance).

Is this the same thing that @eickenberg was proposing above? On my first read, I saw this as allowing sizes to change at runtime, which sounds like a huge task.

> 4. Lastly there is the question on how Kymatio vectorizes computations under the hood. Because we don't cache our tensors nor perform any in-place operations, this can be changed overnight without the user realizing it, so as you say it is indeed a luxurious situation.

Yes. We want to put ourselves in as flexible a position as possible to allow this down the road.
Also, while we're here, 3D doesn't compute the zeroth order. Can we add this in? Currently, there's another function `compute_integrals` which does this (`test_scattering3d.py` uses this to get the full transform and compare it with the stored data).
yes, 3-D should include the zeroth order.
This can be done in a unified API by defining a list `orders` (defaulting to `range(1+max_order)`) in the `Scattering*D` instantiated object.

Working on the tuple vs tensor inconsistency at the moment.


Resolving patch:
diff --git a/examples/3d/scattering3d_qm7.py b/examples/3d/scattering3d_qm7.py
--- a/examples/3d/scattering3d_qm7.py
+++ b/examples/3d/scattering3d_qm7.py
@@ -155,11 +155,9 @@ def compute_qm7_solid_harmonic_scattering_coefficients(
         Returns
         -------
         order_0: torch tensor
-            array containing the order 0 scattering coefficients
-        order_1: torch tensor
-            array containing the order 1 scattering coefficients
-        order_2: torch tensor
-            array containing the order 2 scattering coefficients
+            array containing zeroth-order scattering coefficients
+        orders_1_and_2: torch tensor
+            array containing first- and second-order scattering coefficients
     """
     cuda = torch.cuda.is_available()
     grid = torch.from_numpy(
@@ -208,61 +206,56 @@ def compute_qm7_solid_harmonic_scattering_coefficients(
         full_density_batch = generate_weighted_sum_of_gaussians(
                 grid, pos_batch, full_batch, sigma, cuda=cuda)
         full_order_0 = compute_integrals(full_density_batch, integral_powers)
-        full_order_1, full_order_2 = scattering(
+        full_scattering = scattering(
                 full_density_batch, order_2=True, method='integral',
                 integral_powers=integral_powers)
 
         val_density_batch = generate_weighted_sum_of_gaussians(
                 grid, pos_batch, val_batch, sigma, cuda=cuda)
         val_order_0 = compute_integrals(val_density_batch, integral_powers)
-        val_order_1, val_order_2 = scattering(
+        val_scattering= scattering(
                 val_density_batch, order_2=True, method='integral',
                 integral_powers=integral_powers)
 
         core_density_batch = full_density_batch - val_density_batch
         core_order_0 = compute_integrals(core_density_batch, integral_powers)
-        core_order_1, core_order_2 = scattering(
+        core_scattering = scattering(
                 core_density_batch, order_2=True, method='integral',
                 integral_powers=integral_powers)
 
 
         order_0.append(
             torch.stack([full_order_0, val_order_0, core_order_0], dim=-1))
-        order_1.append(
-            torch.stack([full_order_1, val_order_1, core_order_1], dim=-1))
-        order_2.append(
-            torch.stack([full_order_2, val_order_2, core_order_2], dim=-1))
+        orders_1_and_2.append(
+            torch.stack(
+                [full_scattering, val_scattering, core_scattering], dim=-1))
 
     order_0 = torch.cat(order_0, dim=0)
-    order_1 = torch.cat(order_1, dim=0)
-    order_2 = torch.cat(order_2, dim=0)
+    orders_1_and_2 = torch.cat(orders_1_and_2, dim=0)
 
-    return order_0, order_1, order_2
+    return order_0, orders_1_and_2
 
 M, N, O, J, L = 192, 128, 96, 2, 3
 integral_powers = [0.5,  1., 2., 3.]
 sigma = 2.
 
-order_0, order_1, order_2 = compute_qm7_solid_harmonic_scattering_coefficients(
+order_0, orders_1_and_2 = compute_qm7_solid_harmonic_scattering_coefficients(
     M=M, N=N, O=O, J=J, L=L, integral_powers=integral_powers,
     sigma=sigma, batch_size=8)
 
 n_molecules = order_0.size(0)
 
 np_order_0 = order_0.numpy().reshape((n_molecules, -1))
-np_order_1 = order_1.numpy().reshape((n_molecules, -1))
-np_order_2 = order_2.numpy().reshape((n_molecules, -1))
+np_orders_1_and_2 = orders_1_and_2.numpy().reshape((n_molecules, -1))
 
 basename = 'qm7_L_{}_J_{}_sigma_{}_MNO_{}_powers_{}.npy'.format(
         L, J, sigma, (M, N, O), integral_powers)
 cachedir = get_cache_dir("qm7/experiments")
 np.save(os.path.join(cachedir, 'order_0_' + basename), np_order_0)
-np.save(os.path.join(cachedir, 'order_1_' + basename), np_order_1)
-np.save(os.path.join(cachedir, 'order_2_' + basename), np_order_2)
+np.save(os.path.join(
+    cachedir, 'orders_1_and_2_' + basename), np_orders_1_and_2)
 
-scattering_coef = np.concatenate([np_order_0, np_order_1, np_order_2], axis=1)
+scattering_coef = np.concatenate([np_order_0, np_orders_1_and_2], axis=1)
 target = get_qm7_energies()
 
-print('order 1 : {} coef, order 2 : {} coefs'.format(np_order_1.shape[1],
-                                                    np_order_2.shape[1]))
 evaluate_linear_regression(scattering_coef, target)
diff --git a/kymatio/scattering3d/scattering3d.py b/kymatio/scattering3d/scattering3d.py
--- a/kymatio/scattering3d/scattering3d.py
+++ b/kymatio/scattering3d/scattering3d.py
@@ -340,9 +340,11 @@ def forward(self, input_array, order_2=True, rotation_covariant=True,
                 s_order_2.append(torch.cat(s_order_2_l, -1))
 
         if order_2:
-            return (torch.stack(s_order_1, dim=-1), 
-                    torch.stack(s_order_2, dim=-1))
-        return torch.stack(s_order_1, dim=-1)
+            return torch.cat(
+                [torch.stack(s_order_1, dim=-1),
+                torch.stack(s_order_2, dim=-1)], 1)
+        else:
+            return torch.stack(s_order_1, dim=-1)
 
 
     __call__ = forward


Test patch:
diff --git a/kymatio/scattering3d/tests/test_scattering3d.py b/kymatio/scattering3d/tests/test_scattering3d.py
--- a/kymatio/scattering3d/tests/test_scattering3d.py
+++ b/kymatio/scattering3d/tests/test_scattering3d.py
@@ -74,8 +74,9 @@ def test_against_standard_computations():
         else:
             x = x.cuda()
         order_0 = compute_integrals(x, integral_powers)
-        order_1, order_2 = scattering(x, order_2=True,
-                                method='integral', integral_powers=integral_powers)
+        orders_1_and_2 = scattering(
+            x, order_2=True, method='integral',
+            integral_powers=integral_powers)
 
 
         order_0 = order_0.cpu().numpy().reshape((batch_size, -1))
@@ -83,23 +84,18 @@ def test_against_standard_computations():
         end = order_0.shape[1]
         order_0_ref = scattering_ref[:,start:end].numpy()
 
-        order_1 = order_1.cpu().numpy().reshape((batch_size, -1))
+        orders_1_and_2 = orders_1_and_2.cpu().numpy().reshape((batch_size, -1))
         start = end
-        end += order_1.shape[1]
-        order_1_ref = scattering_ref[:, start:end].numpy()
-
-        order_2 = order_2.cpu().numpy().reshape((batch_size, -1))
-        start = end
-        end += order_2.shape[1]
-        order_2_ref = scattering_ref[:, start:end].numpy()
+        end += orders_1_and_2.shape[1]
+        orders_1_and_2_ref = scattering_ref[:, start:end].numpy()
 
         order_0_diff_cpu = relative_difference(order_0_ref, order_0)
-        order_1_diff_cpu = relative_difference(order_1_ref, order_1)
-        order_2_diff_cpu = relative_difference(order_2_ref, order_2)
+        orders_1_and_2_diff_cpu = relative_difference(
+            orders_1_and_2_ref, orders_1_and_2)
 
         assert order_0_diff_cpu < 1e-6, "CPU : order 0 do not match, diff={}".format(order_0_diff_cpu)
-        assert order_1_diff_cpu < 1e-6, "CPU : order 1 do not match, diff={}".format(order_1_diff_cpu)
-        assert order_2_diff_cpu < 1e-6, "CPU : order 2 do not match, diff={}".format(order_2_diff_cpu)
+        assert orders_1_and_2_diff_cpu < 1e-6, "CPU : orders 1 and 2 do not match, diff={}".format(orders_1_and_2_diff_cpu)
+
 
 def test_solid_harmonic_scattering():
     if backend.NAME == "skcuda":