Repository: xonsh/xonsh

Issue:
Re-thinking the command substitution operators
# Abstract

This is a proposal for re-designing the command substitution (a.k.a command pipelines) mechanisms in xonsh.

A simple example for command substitution in bash:
```bash
echo hi $(whoami)
```

#### NOTE: This example is very basic and naïve. Please look at the usage examples below.

# Motivation

Command substitutions have a range of usage scenarios, and we want them all to be easy and concise.
These usages differ by two factors - stripping and splitting.

* Stripping: Most unix and windows cli commands append newlines to their output, which we might want to strip.
* Splitting: A program might output multiple values separated by spaces, newlines, tabs or other tokens, which we might want to split to multiple arguments.

#### Some of the more common combinations aren't easily expressed in xonsh currently.

Examples for each scenario (commands with the corresponding output format):

|                          | Strip trailing newline | No stripping | Strip all whitespace |
| ------------------------ | ---------------------- | ------------ | -------------------- |
| No splitting             | `whoami`, `uname`, `hostname`, `nproc` | `echo -n ... \| wc -l` | |
| Splitting by newline     | `find`, `git branch/log/diff --name-only` | | |
| Splitting by whitespace  | _Meaningless_ | _Meaningless_ | `groups`, `yay -Qdtq`, `pkg-config` |
| Splitting by other token | `parsing /etc/passwd` | | |

## Current xonsh behavior

| Operator | Strip | Split | Return value | Mode |
| -------- | ----- | ----- | ------------ | ---- |
| `$()` | No | No | String | Both |
| `!()` | No | No | CommandPipeline object | Python |
| `@$()` | Whitespace | Whitespace | List of args | Command |
| `$[]` | No | No | None | Both |
| `![]` | No | No | CommandPipeline object | Both |

## Current state and comparisons with other shells

The **first three** options are the more common ones according to our current examples.

| Strip                 | Split          | Xonsh | Bash | Zsh | Fish |
| --------------------- | -------------- | ----- | ---- | --- | ---- |
| **Trailing newlines** | **No**         | `@($(CMD).rstrip('\n'))` | - | `"$(CMD)"` | - |
| **Trailing newlines** | **Newlines**   | `@($(CMD).splitlines('\n'))` | - | - | `(CMD)` |
| **Whitespace**        | **Whitespace** | `@$(CMD)` | `$(CMD)` | `$(CMD)` | `(CMD \| string split ' ' --no-empty)` |
| No                    | No             | `$(CMD)` | - | - | - |
| Whitespace            | No             | `@($(CMD).strip())` | `"$(CMD)"` | - | - |
| No                    | Newlines       | `@($(CMD).split('\n'))` | - | - | - |
| Whitespace            | Newlines       | `@($(CMD).strip().split('\n'))` | - | - | `(CMD \| string trim \| string split ' ' --no-empty)` |
| Trailing newlines     | Token          | `@($(CMD).rstrip('\n').split(TOKEN))` | - | - | `(CMD \| string split TOKEN --no-empty)` |
| No                    | Token          | `@($(CMD).split(TOKEN))` | - | - | - |
| Whitespace            | Token          | `@($(CMD).strip().split(TOKEN))` | - | - | `(CMD \| string split TOKEN --no-empty \| string trim)` |

#### NOTE: As far as I can tell, none of the shells has an option to only strip the **single** last newline.

# Current proposals

## [XEP-2](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst) (by @anki-code)

See details and examples in the XEP - https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst.
In short:

Mode | Current | Proposed
-- | -- | --
Python | $() returns output string. | $() returns CommandPipeline object.
Subproc | $() returns output string. | $() returns list of lines from output without trailing new line symbol.
Python | $[] returns None. | $[] returns HiddenCommandPipeline object.
Both | !() exists. | ~!() removed.~ UPD: discussion.
Both | ![] exists. | ~![] removed.~ UPD: discussion.

`CommandPipeline` (CP) class changes:

* Add str representation the same as `CP.out`.
* Remove trailing new lines in `CP.out`, `CP.lines` and `CP.__iter__`.
* Add all string methods i.e. the `$().split()` will return `CP.out.split()` that is IFS analogue in fact.
* Add all string methods for lines i.e. `$().lines_find(txt)` will return `[l.find(txt) for l in CP.lines]`.

## This issue's initial proposal with influence from @adqm

* Strip trailing newline in `$()`.
* Strip trailing newlines in `!().__iter__` (i.e. use `.splitlines()`).

### Pros:

* No type-changing back-compatibility issues.
* `!()` is consistantly iterated through.
* Splitting by whitespace remains unchanged (`@$()`).
* The complete output is still available easily via `!().out`.

### Cons:

* Python code that uses `$()` might need to change if the trailing newline is important.

## https://github.com/laloch/xonsh/commit/98363df (by @laloch)

Add `@$[]` operator to split by newlines.

### Pros:

* Completely backwards-compatible.

### Cons:

* More arbitrary syntax.
* Doesn't really extend `@$()` and `$[]` in a meaningful way.
* Still doesn't solve the most common `No split, Strip newline` scenario.

# Rejected proposals

## Initial proposal in this issue

Run `.rstrip('\n')` in `$()`.

### Cons:

* No easy way to use the complete output if a user needs it.

## Add a config option to strip `$()`

### Cons:

> this is a case where configuration would be bad, because xonsh scripts would either run or fail based on how the environment was configured.

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**



Discussion:
Hi @daniel-shimon, this has been discussed (and rejected) several times already. The reasoning, if I remember correctly, was that the trailing newline is part of the captured output and therefore should be kept as is. What if the terminating newline character is significant to your pipeline? For instance `echo "foo" | wc -l` vs. `echo "foo\n" | wc -l`? What about outputs terminated by several newline characters? How many of them would you want to strip?  `rstrip(...)` strips all of them...

Btw, I use `git rebase -i @$(git merge-base HEAD master)` several times a day and it does its thing just fine :wink:

There's a little trick using `abbrevs`: You can define `abbrevs["$$"] = '@($(<edit>).rstrip("\\n"))'`.
I understand what you're saying but it still doesn't feel very helpful. I'm talking of course only about subprocesses which are substituted inside another command, in which case I can't really see how that's helpful.

In any case I guess this issue is a pretty opinionated thing if it has been discussed so many times.
Do you think adding a config option like $XONSH_STRIP_SUBPROC would be acceptable? 
> In any case I guess this issue is a pretty opinionated thing if it has been discussed so many times.

Yeah, it really is.  The fundamental issue is that many CLIs add a trailing newline without detecting if they are connected to a TTY or not. Bash and other shells effectively add an rstrip operation when capturing (or really a split operation), effectively deciding what the output of subprocesses should be.  

I personally think that a lot of Bash's syntax and magic is particularly confusing to new users. 

> Do you think adding a config option like $XONSH_STRIP_SUBPROC would be acceptable?

I wish it was. I think this is a case where configuration would be bad, because xonsh scripts would either run or fail based on how the environment was configured. It seems like it could cause a lot of downstream bugs. So I think that this is a scenario where a strong decision is needed. 

> I'm talking of course only about subprocesses which are substituted inside another command

Now this is an interesting idea.  I am open to considering it.  To-date all operators work exactly the same independent of their calling context.  Things that would need to be fleshed out for this proposal.

* What happens with the other subprocess operators depending on their calling modes: `![]`, `!()`, `$[]`
* What do we do with the `@$()` operator?

I do like to keep an open mind, though! So I can be convinced away from the above opinions. 
@scopatz what if create distinct operator for this?
> what if create distinct operator for this?

That is what `@$()` was supposed to be :wink: I am in favor or removing syntax at this point, rather than adding. I really want to avoid the issues that Bash has where there are so many specialized syntax elements that no one can remember them. Xonsh should be easy for people to learn and use and consistent in its implementation. 

If anything I think we should either fully implement `@$()` or remove it.  The initial idea for `@$()` what that you could register transformation functions (like a decorator), that would modify output.  For example, `@upper$()` would uppercase the output.  Or you could apply many times, like `@split@upper()`.  Then what we have now would just be the default value: `@split$() == @$()`. 

However, this was never fully done, so maybe it is better to drop the syntax entirely.
> I do like to keep an open mind, though! So I can be convinced away from the above opinions.

Good to hear! I was fearing this is a dead end.

1. I agree with @anki-code that `@$()` should remain as-is since it's very xonsh-specific in nature and the ability to spread arguments is pretty cool
2. I think the other subprocess operators shouldn't change as well, since if we're using them we're probably doing something more advanced than just command substitution.

The basic jist is that `$()` is used daily for command substitution, and keeping the trailing newlines isn't robust enough for day to day use (e.g. when dealing with paths you have to use `@$()` which is cumbersome and won't work for paths with spaces).

If we're using `$()` in more complex python code, we probably want to be accurate, and the command line(s) will probably be bigger such that adding a `.rsplit()` is more legitimate
I already have `@$[]` operator here long forgotten, that splits the captured output on newlines only, yielding
```
$ showcmd @$[echo "space here\nbreak here"]
['space here', 'break here']
```
instad of
```
$ showcmd @$(echo "space here\nbreak here")
['space', 'here', 'break', 'here']
```
I can open a PR if there's general interest in the functionality.
I do, however, agree with @scopatz in that we should not add more syntax for no good reason.
@laloch could you please share your PR? I want to understand the process of creating additional operators =)
Another option is to only have `$()` strip the trailing newlines, while `!()` doesn't.
This way `$()` is robust for day-to-day command substitution but for full accuracy one can use `!().out`.

This resonates with `$()` being intended for simple usages (you can't use it to get the exit code etc), so I think adding the `rsplit` to make it more robust and useful is pretty sound
Yeah I agree about not adding more syntax in this area
> could you please share your PR? I want to understand the process of creating additional operators =)

Yeah, of course. Here you go: laloch/xonsh@98363df. It's pretty trivial, since it copies almost everything from `@$()`.
> I like this logic when I thought about users from bash world and fast diving into xonsh. Looks consistent.

It's not just for users coming from a different shell, it's really more useful and intuitive since you want information from some standard unix command (e.g. whoami, uname, nproc etc), and they all append newlines to their outputs.

Here's another example from a classical daily shell use - `make -j $(nprocs)` fails. Yeah, you could use `@$()` but it doesn't really express what you want (stripping, not splitting), and I don't think this kind of simple classic shell usage should cost a lot of syntax overhead.
I have only one concern #3394 :)
#3394 is a bug - sometimes you need to call `!(cmd).end()` in order to get the resulting `CommandPipeline` attributes populated.
My main point here is that if we are going to make one subprocess operator modal (ie Python vs Subprocess), it is worth thinking about how all of them should behave in a modal fashion.

Also, if as part of this, we should decide if we can get deprecate `@$()` or if we should fully implement it, and how this operator would behave modally too.

@anki-code's suggestion [here](https://github.com/xonsh/xonsh/issues/3924#issuecomment-715625964) gets close to that, but is probably not the complete answer.

I guess what I want to see is a table with all 5 subprocess operators and how they would behave (ie what their return values would be) in both Python and subprocess mode. I think this would be a good exercise because such a tutorial would need to make it into the docs eventually anyway, if we proceeded down this path.
The PR - https://github.com/xonsh/xonsh/pull/3926
@anki-code if I understand correctly with your suggestion `$()` isn't modal?
Meaning `$()` always returns the rstripped output, and `!()` returns the complete output in subprocess mode and a CommandPipeline object in python mode?

This sounds like a pretty good idea to me - `!()` is used for fine-grained usages and `$()` just works for substitutions.
@scopatz In this suggestion, the `!()` operator doesn't really **act** differently wrt to the current mode, it just returns a helpful thing in subprocess mode and returns the complete `CommandPipeline` object in python mode.
I've edited the description of the solution.
@anki-code very nice!
Can you clarify that the behavior should be `.rstrip('\n)` like in other shells?
Also can you add the `make - j $(nproc)` example? 
In my opinion, `$()` should return a string in both Python & subprocess mode, not a list. The other operators are free to return various other objects
@scopatz could you give an arguments why? 
Anyway, on the technical side, the original design was for `$` to indicate that a built-in Python object was returned (str or None), while `!` indicated a more rich object, now `CommandPipeline`. `()` indicated capturing, ie nothing would be printed to stdout. `[]` means that output flows through to the screen.

So in this light, `$()` means capture the output and return a basic Python object. It is supposed to be the same as a the Bash operator `$()`.

Now, I think there might be some confusion about what Bash does. In Bash, because everything is a string*, the `$()` just returns a string of stdout.  I am not even sure that Bash strips the trailing newline.  However, in most Bash usage, [the Internal Field Separator `$IFS` variable](https://bash.cyberciti.biz/guide/$IFS) actually determines how text strings are split by the Bash parser. By default this is set to do whitespace stripping. So in effect, `$()` in Bash works like `@($(cmd).split())` would in xonsh.

In my opinion, `$IFS` is a bad idea because it can lead to all sorts of strange behavior, like 3rd party scripts not running because the calling process has set this variable incorrectly. `$IFS` also makes `$()` seem like it is doing more than it is. 

I think using `$()` in xonsh to split into a list of arguments is a neat idea, but it would necessitate the addition of some default or configurable way to split those arguments. For example, should `$()` be split by lines or by whitespace (like effectively what Bash does)?

Like with Bash, splitting `$()` into a list of str and being able to configure how this split happens would have the following issues:

* Some scripts would not be able to run with each other or 
* Every script would need to be very aware of how `$()` was being split. 

This is why I think that `$` simple return operators like `$()` and `$[]` should pick one uniform way of returning and stick to that. It builds a consistent ecosystem of code that can be run with each other.  The approach in xonsh so far has been to thus allow the users / authors to explicitly split however they want. `@($(cmd).split())` may seem like more work to write, but it is less work to inspect and debug later on. 

So while I keep an open mind on what these various operators should return, I think we should also try to learn from some of the bad design of other shells.

I hope this makes sense!

*Yes, you can make arrays, but it is hard and annoying to do so.

----

On the less technical side, I think there are a lot of things that we all want fixed before 1.0.  This is another problem that probably should be resolved (one way or the other). 

Speaking only for myself, xonsh is something that I can only code on during the nights and weekends. I have other things to do in my free time as well. Because I am trying to support all parts of xonsh (review all PRs, put out releases, fix fundamental issues with the project, etc), I don't have as much time as even I would like to spend on each issue. 

There are other forks of xonsh out there.  Which is great! Xonsh has made the process of creating high quality shells so much easier, that I am happy that people have made improvements in the direction that they want to go. Many years worth of effort have gone into xonsh from a fairly big group of people.

I am sorry if you feel development is too slow. It is going as fast as it possibly can. An single person working on their own is always able to work faster than a group trying to make decisions.

On the other hand, working to improve xonsh as a group, even though it is harder and slower, does improve the shell for everyone in the long run. If a change is important and valuable, it will make it in. But the bigger the effects of the change, like with subprocess syntax, the more people are going to need to weigh in on the pros/cons and the more forethought will need to be put into it.  
I agree we should improve the efficiency of the discussion, and document all the arguments in a centralized way.

Following what you guys said, since this is such a major design decision in xonsh I suggest we take a step back from implementation details, gather concise examples, compare to other shells and document the pros-and-cons for each proposal.

I'll try to make this issue a mini-PEP for the subject.
@scopatz if you prefer some other medium, a separate repo could be cool.

I've updated the description to express all the things we've talked about. 
@anki-code @laloch please let me know if I missed stuff and give us more examples so that I'll update the description.

Let's take a few days to get more examples, discuss more options and let more people from the community express their opinions :smile:
Chiming in since I was one of those who was spam-mentioned in the other thread and because I was involved in the discussion and implementation of these operators the first time around, though I've not had a horse in this race for several years...

(BTW, hi again, @scopatz et al!  It's been a long time; I hope all is well!)

-----------------

Years ago when this conversation started, I was a proponent of stripping a trailing newline from the output of `$(...)` but leaving the output as a string; and leaving `!(...)` as-is.  And I'm still a proponent of that change, FWIW (was that the original proposal in this thread?  I haven't been following the conversation).

I'm not crazy about the implementation in #3926, since `$(...)` returning a list just doesn't feel right to me.  You can always do `list(!(...))` if you want something like that.  I'd rather see fewer keystrokes for common operations, and more keystrokes for less common things, so I think it makes the most sense for `$(...)` to return a string with trailing newline stripped, and `!(...)` to return whatever fancy object xonsh returns nowadays for that kind of operation.  So I would like to see:

```
$ user = $(whoami)

# easy, and common, and (i'm pretty sure) matches bash in terms of what is captured...
# if not, it at least matches my expectation and doesn't surprise me when i switch between shells
```
```
$ user = !(whoami).out 

# if you want the trailing newline for some reason
```
```
$ files  = list(!(ls))

# if you want a list instead of a generator for some reason
```

This is the behavior I have in my fork, and I'm happy with it; I've never been surprised by what I get out of either of these operators, and examples like the `make -j $(nproc)` work just fine.  I have never used the middle example (with the trailing newline), though I do loop over `!(...)` directly regularly, and I occasionally make a list out of it (as above) for later use.

--------------

I feel like this small change is also all that one needs to provide a relatively easy way to get just about anything you would want from a command.  In Python mode:

* If you want the output as a string, you can do `$(...)`, no need for any indexing, stripping, joining, etc
* If that deleted some whitespace that you _wanted_, you have to type a bit more, but you can do `!(...).out`
* If you want something to loop over the lines, you can do `!(...)`

Then, in terms of interpreting as arguments to a subprocess-mode command:

* If you want the whole result as a single argument, you can do `command $(...)`
* If you want the result as multiple arguments split by whitespace, you can do `command @$(...)`
* If you want the result as multiple arguments split by newlines, you can do `command @(!(...))`

I feel like this structure, on the whole, is less work for the more common operations than the current behavior, and also when compared to the structure where `$(...)` returns a list-like thing.  And if you want something more complicated than the above, I feel like you should be able to build it from those primitives.

Basically, I feel like the current behavior is mostly consistent, and that the single small change of r-stripping newlines by default in `$(...)` reduces end-user surprise.  I'm not sure the additional changes of the structure proposed in #3926 provide much additional benefit, particularly when weighed against the cost of such a big (and backwards-incompatible) change.

-------------------------

I don't have the time or the energy to jump into the fray here, though, other than to say that I am probably -1 on #3926 if I still get a vote, to agree with @daniel-shimon that these operators are indeed worth thinking about more, and to present my view of how I would want things to work.
@scopatz I am in favor of removing `@$`. We could find some pythonic ways to do it instead. Having too many operators leads to confusion. Python is as simple as one can get. We could use/abuse the pipe operator https://pypi.org/project/pipe/ 😄. We can discuss that in another issue I think. 

We can blame bash for all we want but it is everywhere, even dockerfilss, CI configs etc, Copying the syntax but not the behaviour leds to confusion and frustration. I myself copy pasted code and expected it to run (my bad it was in bash😄). I understand that Xonsh wants to establish high standards. But come on, bash is not going to be replaced anytime soon and xonsh is primarily interactive usage (Though the scripts can be in xonsh, I find myself using plain python+[sh](https://pypi.org/project/sh/). It is way more maintainable and has complete toolchain to lint,format etc.,). So why not make it more usable and less surprising.

@anki-code Having said that, it is not fair to add such disruptive behaviour to an old project and have it by default. Many people would be having their scripts doing things using this behaviour. The best bet would be to make this a xontrib. Since this is a core code, making it modular, maybe it can be achieved.
> Having said that, it is not fair to add such disruptive behaviour to an old project 

@jnoortheen thanks for your attention and time! The backwards compatibility is not a strong argument and xonsh is not so old (5 years slow xonsh development vs 15 years of fast fish development vs 30 years zsh/bash). Bash becomes ugly because thinks about backwards compatibility every day.
Hi @adqm, thanks for the detailed explanation!
A few questions:

> I think it makes the most sense for $(...) to return a string with trailing newline stripped

1. Do you mean to exclusively strip **one** trailing newline or all of them?

> If you want the result as multiple arguments split by newlines, you can do command @(!(...))

2. Currently each line returned here contains it's trailing newline. In light of this discussion, don't you think these newlines should be stripped as well (like `str.splitlines()`)?

> If that deleted some whitespace that you wanted, you have to type a bit more, but you can do !(...).out

3. What about having `!()` return the entire output in command mode? I personally think it's inconsistent with `@(!@())`, but I want to hear your opinion.

#### I've added this proposal with the changes I think are necessary (strip newlines in iteration), feel free to improve on it

@anki-code I updated the description 👍 

On another note:

> > Returning a list in python mode beats the simple intuitive meaning of $().
> 
> Absolutely no. The `$()` is a substitution operator in other shells and it's absolutely expected that `$()` will substitute arguments list. 

What I mean is that the splitting by **lines** is unintuitive and unfamiliar for the default `$()` implementation.
Most of our examples come from the upper-left corner of the usage scenario - `No split, Strip newline`.

In this regard, I feel that your proposal does too much magic for these cases. You're splitting by lines even though that's not what a user **intends** to do when using commands like `uname`.
The PR **assumes** that these commands will always contain one line, so the user's intent will align with `OutputLines` behavior.

This is the reason as of now I agree with @adqm the most. 
We already have line splitting with `!().__iter__`.
If it will strip the trailing newlines (e.g. use `.splitlines()`), in conjunction with `$()` stripping the trailing newline we will easily have all three major use cases:
* Strip trailing newline, **no** split `$()`
* Strip and split by newline `@(!())`
* Split by whitespace `@$()`
@daniel-shimon I reviewed the description and I've found that the table "Strip | Split | Xonsh" is unclear. It just hide the examples and use cases. Examples added to [XEP-2](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst).
Honestly the specifics of this is beyond me right now, but I'm glad people are working through it. Keep up the good work!
@daniel-shimon I dived into the `@(!())` example and found new (for me) strange behavior of the current xonsh version operator:
```python
list(!(head -n3 /etc/passwd))
# ['root:x:0:0:root:/root:/bin/bash\n',
#  'daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin\n',
#  'bin:x:2:2:bin:/bin:/usr/sbin/nologin\n']
```
The `CommandPipeline` class is iterable and returns list of lines with trailing new line symbol (that should be stripped again if we want to use it as arguments 🤦). This strange behavior should be mentioned in description.
I've read twice the @scopatz [comment](https://github.com/xonsh/xonsh/issues/3924#issuecomment-716180869). Resume:

UPD: https://github.com/xonsh/xonsh/issues/3924#issuecomment-723510857
@anki-code 

> But you mean under this sentence - different from current xonsh behavior

You're right, changed it.

> In bash if you call $(uname) the result will be the same as in my PR.

Agreed.
What I'm saying is that if we think about the real intent when using commands like `uname`, `hostname` etc, you want the **content**, not lines. This is why they're in the `No split` section in the examples.
The fact that their output has an extra newline is because unix tools always append it regardless of the tty. It doesn't mean you want to split them by lines.

> The meaning of this two point is similar. Please unite it into one.

They're two different things:
1. `OutputLines` is a new thing users need to learn.
2. The fact that sometimes it works like a string makes it a really error-prone subtelty in my opinion. Since this is more of a feeling than a backed-up clame I'll remove it :)

> We need a copy-pastable pack of examples (use cases) that show how every approach works

I prefer this format.
There's one table that shows all the examples we have, and a table for each proposal with their solutions per scenario.
It's a bit less clean but provides the entire scenario. I'll try to make it a bit more readable though :star:

> Also you completely miss the logic from my PR about consistent operators

I don't, it's in the table in your proposal.

> Maybe It's a good idea to show how sh-shells solve the same use cases

I do, see `Current state and comparisons with other shells`

> Please mark this bullet point by prefix "[Reported to PR author, waiting for confirmation or solution]".

This whole thing is just a draft, and I'll update the proposal when you'll come up with a solution, promise.

> Is not working cute in subproc mode. You should do du -sh @($(ls).__iter__) (or similar). 

Whoops, typo!
That was meant to be `!().__iter__`, meaning iterating over `CommandPipeline` like you saw happening with `@(!())` and `list(!())`.

> The CommandPipeline class is iterable and returns list of lines with trailing new line symbol (that should be stripped again if we want to use it as arguments 🤦). This strange behavior should be mentioned in description.

It is - that's what I meant in the second bullet of adqm's approach (the typo above).

As of now, I have two issues with @anki-code 's proposal:

* No clean solution for `no split, strip newline`. For me at least, this seems to be the daily usage.
* I'm not convinced the custom `OutputLines`'s benefits are really worth it. In my experience at least, customized hybrids like this do more damage than good in the long run. They're hard to maintain, they surprise users and require a large learning curve.

Then again, I don't have a lot of experience in the **shell** business 😸 
@scopatz I'll appreciate your view on these issues if you have the time.
I don't have a super strong opinion here, these are just my thought in the current state of things.
Hey all, my thoughts generally align with @adqm (Hi! Hope you are doing well too!). Other than that, I think I am going to take a week break from this issue. I don't have the bandwidth to follow all of the details of the discussion. I suggest everyone also take a small break from this particular thread as well.  
@daniel-shimon and JFYI @scopatz (because you like adqm solution too)

The cons of the adqm approach that current tables hide:

#### First ➖ 

As we discuss above default `$()` behavior in other shells is to pass splitted list of arguments to the command. If we have the `file1\nfile2\nfile with spaces\n` output then in bash it will be the list of arguments `['file1', 'file2', 'file', 'with', 'spaces']` ($IFS defines split symbols in bash). 

In the adqm approach you just remove the triling new line and got the `file1\nfile2` - this is: a) not common and whell known behavior for the shells and b) not working with many arguments (i.e. `du -sh $(ls)`). In fact you try to solve only one use case - when the output have only one line and you hide in your table that many lines use case is not solved (you need to do additional syntax to it `@(!())`).

As result the statment for adqm solution `The three most common cases are short and concise.` - looks strong but have questions about consistency behind and we can only talk about "shorter syntax". This pros bullet point should be: `"The three common cases have shorter syntax in the command."`. 

*This lack of info produced by reducing use cases into "Strip | Split | xonsh" table.* If you want to create full matrix you should add additional columns to the table: 1. mode (subproc, python), 2. count of lines in output (one, many), (optionally: 3. what if we need custom delimiter (i.e. `-` instead of `\n`) ).

In XEP-2 both cases are solved: `cat /etc/passwd | grep $(whoami)` (single line) works and `du -sh $(ls)` (many lines) works.

#### Second ➖ 

The second thing that is can go to cons in adqm solution is that it tries to replace common well known `$()` behavior (to pass splitted list of arguments to the command) from other shells to new `@(!())` syntax construction. Instead of using `$()` as list of arguments by default (like in other shells and XEP-2) this approach creates unfamiliar `$()` behavior (stripping trailing new line and return output) and suggest the alternative like `@(!())`. 

As result the commands like `du -sh $(ls)` will work when `ls` has one file but the command will fail when `ls` will have many files and the command should be rewritten as `du -sh $(!(ls))` - that is very unclear, uncommon, unexpected and require tons of clarifications why it works like this. 

For what this mess with behavior and syntax in comparing to another shells? This is cons.

In XEP-2 `du -sh $(ls)` will work forever and as expected.

#### Third ➖ 

Anthony wrote:

> **I think using $() in xonsh to split into a list of arguments is a neat idea**, but it would necessitate the addition of some default or configurable way to split those arguments. For example, should $() be split by lines or by whitespace (like effectively what Bash does)?

This question is ignored in adqm approach because `$()` in this approach will remove trailing new line and return output forever and we will faced with new questions if we want to add custom arguments delimiter. This is to cons.

Instead of this in XEP-2 `$()` already returns the list of arguments by design. We free to allow to set custom delimiter and it will not something new for user. We just tell user "you can set dash instead of new line as delimiter". User answer "ok, thanks :)". My PR has long term and agile logic  instead of adqm approach.

*PS: I'm not the English naitive speaker and there is lack of idioms in my text. This is the cause of my text could looks angry. Please make the discount :)*
I've read messages above again (from scopatz, adqm, daniel-shimon, laloch) and I want to do some meaning points. Guys, when you reasoning you miss some prerequisites:

1. **There is python and subproc mode** and it sounds obvious but when you talk about pros of one approach in python mode you miss the cons in subproc mode. But subproc mode is less agile than python and requires more attention.

2. **There are cases with one line output and many lines output**. When you focus on pros for "one line output" solutions you miss the behavior in many lines.

3. **The `$()` operator was built [to pass arguments list since 1979](https://en.wikipedia.org/wiki/Command_substitution)** in other shells. The xonsh behavior that returns pure output in this operator - is not common and well known. "Command substitution" article in Wikipedia: ` ... reads that output, parsing it into [words] separated by [whitespace]`.

4. **We have "game-changers"**: 
    a) the idea of different behavior of the operator in subproc and python mode, 
    b)  the Python class ability to return list or str representation from the object, 
    c) the ability to add any functions to `!()` object. This makes operators agile and we don't need to invent the functionality that spread around all operators we have. We can just focus complex functionality in one `!()` operator and leave the old man `$()` returns just the list or string representation.

5. **Wrong examples with `.rstrip('\n')`**. The `rstrip` function will remove all trailing `\n` from the line. The output `line\n\n\n` becomes `line`. It looks ok on first look but it's wrong, magically and not consistent (i.e. with meaning point 2). This kind of modifiers should be custom not common. For right split-stripping we already have Python built in solution: `str.splitlines()` that used both to split strings and read lines from the files in Python.

6. **In the tables that describe approaches there is only "Trailing newlineS" examples**. The example with removing one trailing new line is missed.

7. **Backward Compatibility is the killer** when bad design exists. The older xonsh is, the more toxic the backward compatibility will be. When we imagine great xonsh in the future the `@($().rstrip().split('\n'))` spells and stripping calls in every command don't look great and modern.

Please, when you try to reasoning don't give us just words. Show us the full solution: subproc and python, one line output vs many lines, `$()`  and `!()` behavior that consistent with meaning points above.
Folks! I've created the repo for my proposals - [xonsh-operators-proposal](https://github.com/anki-code/xonsh-operators-proposal/blob/main/README.rst)
![image](https://user-images.githubusercontent.com/1708680/97853379-1807af00-1d09-11eb-8333-5cce56edef34.png)

Hello all! Hope your weekend going well.

I made a lot of homework during the week of silence. I took into account all the comments and suggestions in this and other topics. I want to told you about the result.

1. I figured out one new approach that looks the best. I described this approach in distinct page as [Xonsh Enrichment Proposal #2](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst) (XEP-2). I'll not repeat it here just read it.

2. I implemented the XEP-2 approach to allow everyone try it in action.

3. To achieve point 2 I crated [my-xonsh-fork](https://github.com/anki-code/my-xonsh-fork) script that allows completely rename xonsh source code to make new package name.

4. As result I created [xonsh2](https://github.com/anki-code/xonsh2) repository that could be installed via pip, contains XEP-2 approach changes and will not affected to your installed xonsh on the host. 

    You can install xonsh2 and: run it, create `.xonshrc_2`, add to the shebang (`#!/usr/bin/env xonsh2`), run scripts (`xonsh2 my.xsh`), put xontribs to `xontrib2` directory and load it via `xontrib2 load`. Original xonsh will live shoulder to shoulder with xonsh2 in fact.


Finally, I want to ask maintainers and any people who want to help read the XEP-2 proposal, try this approach in action and give a feedback. Feel free to report bugs and features to the [XEP-2 repo](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst). Thanks!

*PS: first feedback on XEP-2:*

<details>

![image](https://user-images.githubusercontent.com/1708680/97809651-2dd09200-1c7f-11eb-815d-0fbe10c19fe3.png)

</details>

After creating [XEP-2](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst) I want to read the @scopatz [comment](https://github.com/xonsh/xonsh/issues/3924#issuecomment-716180869) and answer more detailed. 

> the original design was for $ to indicate that a built-in Python object was returned (str or None), while ! indicated a more rich object, now CommandPipeline

This just an idea. You are not described the benefits behind or arguments why we should support this idea.

We deleted `!` operator in XEP-2 and have the same functionality with better use cases.

> So in this light, $() means capture the output and return a basic Python object. It is supposed to be the same as a the Bash operator $().
> ...
> In my opinion, $IFS is a bad idea because it can lead to all sorts of strange behavior

I agree with arguments against the IFS. But this can not be used as an argument that the `$()` operator should return string. 

Yes, IFS is bad because changes behavior of `$()` in Bash. This only means that we shoud do the `$()` operator stable in xonsh. It could be a string or list or something else but the main requirement is: it should be unchangeable.

> I think using $() in xonsh to split into a list of arguments is a neat idea, but it would necessitate the addition of some default or configurable way to split those arguments. 

Here you contradict yourself. You criticize the IFS above for this. No, we shouldn't change the `$()` behavior externally. We should choose the most usable behavior and stay on it.

> For example, should $() be split by lines or by whitespace (like effectively what Bash does)?

Rephrasing this questions: what `$()` should do by default? So we are here for discuss it. Our current suggestion is in [XEP-2](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst).

> Like with Bash, splitting $() into a list of str and being able to configure how this split happens would have the following issues
> ...
> This is why I think that $ simple return operators like $() and $[] should pick one uniform way of returning and stick to that.

We all already agree that IFS is bad and we should choose the good unchangeable behavior for `$()` by default.

We still not have an arguments why it should be a string.

> The approach in xonsh so far has been to thus allow the users / authors to explicitly split however they want. @($(cmd).split())

This statement tells to us that we should have an ability to do this. But it does not tell to us that we should use `$()` operator for this.

There are four counter arguments against that `$()` operator should be a string in subprocess mode:

1. This operator have [a long and well known history](https://en.wikipedia.org/wiki/Command_substitution) - it used to split output to arguments.

2. The [Zen of Python](https://www.python.org/dev/peps/pep-0020/#id2) told us: `Beautiful is better than ugly`, `Simple is better than complex`, `Flat is better than nested`, `Readability counts`. Now look here: `id $(whoami)` vs `id @($(whoami).rsplit())`. The left command is obviously in line with Python Zen principles. Even if you try to argue using the principle `Explicit is better than implicit`, you will lose to other principles and points 1,3,4 in this list.

3. This operator does not return pure output. The output is modified (read below). You can not refer to use case where user get pure output and split it as he want.

4. There is a technical and in most cases backwards compatible way to make `$()` pretty agile to solve explicitly splitting for complex usage. Proposed in XEP-2.

5. Added later: [Statistics regro/rever](https://github.com/xonsh/xonsh/issues/3924#issuecomment-728351849) shows that getting string use case loses to getting arguments use case.

As result the statement about explicitly splitting still has no arguments about why `$()` should be a string in subprocess mode but has counterarguments.

I want to additionally demonstrate that current `$()` operator corrupts the output:

```python
body = '1\n2\r\r\r\r3\r\n4\n6\r7'

$(echo -n @(body)) == body
#False
len($(echo -n @(body))) == len(body)
#False

$(echo -n @(body))
#'1\n2\n\n\n\n3\n4\n6\n7'

echo -n $(echo -n @(body)) | md5sum
# 8dae88fa7c93e91a59abea2686e64994  -

echo -n @(body) | md5sum
# 35364d7e6a46e5f560e2e9509be31349  -

# Prepared by xontrib-hist-format
```

This means that current `$()` is not pure output. Any arguments that based on that user has an output and he can split it any way are all wrong. User already got corrupted output and should invent the way to do something with this. Current xonsh has no pure logic. Current xonsh is already has biased pre assumption that `$()` should return corrupted output.

Now back to @scopatz [comment](https://github.com/xonsh/xonsh/issues/3924#issuecomment-716170590):

> In my opinion, $() should return a string in both Python & subprocess mode, not a list.

This statement:
* Has no strong support behind.
* Has no arguments behing.
* Has reasonable counter arguments.
* Has reasonable alternative described in [XEP-2](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst).

I commented on Anthony's answer line by line and showed the lack of arguments not to attack him. Anthony is the founder of this incredibly awesome project and his opinion requires special attention. In this case this opinion does not have strong support and I just wanted to show it and urge all of us to what is included in the title of this issue - thinking. 

I'm looking forward to [XEP-2](https://github.com/anki-code/xonsh-operators-proposal/blob/main/XEP-2.rst) review from Anthony and xonsh/xore.
Given XEP-2 says that it is backwards incompatible (which it is) and you want a xonsh2, I don't know how to approach this in any short time frame.
That is to say, is there any way to change this so that it is actionable in the near future? Maybe have something which is mostly backwards compatible, so that we can move toward 1.0 with whatever changes we do want to make.
I think the decision should be left to user, instead of forcing them to adapt to a behaviour one seem fit. 
So making this configurable with some list of choices(string names like `no_new_line`, `raw` ...) and a callback path(to return custom class) would suffice all use cases. 
@scopatz Why is backwards compatibility a big issue before we're at 1.0? If it's because of the time frame maybe we can split tasks to a few parallel PRs? 
Yeah, I 100% agree that we are not at v1.0 yet and so we should feel free to make backwards-incompatible changes.  However, I don't know that we can fundamentally break every xonsh script & module that has ever been written. 

I would also like to avoid Python 2 ->3 transition issues, especially in pre-1.0 world. 

I guess we could use the current parser to translate older APIs into the newer ones, if we did want to go down these route. 
Also, is there a place I should be giving specific feedback on XEP-2?
If you want to ask questions or point out to grey zones that should be described or improved in the proposal the best way is [to create an issue in the repo](https://github.com/anki-code/xonsh-operators-proposal/issues/new). I'll think about it. We already have a question from laloch [that was answered](https://github.com/anki-code/xonsh-operators-proposal/issues/2).

Other thoughts feel free to put in any place you want.

PS: it will be great to know who is from xore team also have an interest in this process and from who we can wait a feedback.
Minute of statistics. I've counted the meaning of `$()` operator in [regro/rever](https://github.com/regro/rever) that written on xonsh:

1. Only 5 lines (30%) use `$()` operator to get string. All other 12 lines (70%) use `$()` to get "arguments" and have `strip()` or `splitlines()` tails.

    <details>

    ```python
    cd /tmp
    git clone https://github.com/regro/rever/
    rm -rf rever/.git
    grep -ri '$(.*)' ./rever | grep -E -v '(appimage|\.rst|/test|filename=|Autogenerated)+'
    ```

    | File | Line | Meaning |
    |---|---|---|
    | `rever/conda.xsh` | `envs = json.loads($(conda env list --json))["envs"]` | str |
    | `rever/conda.xsh` | `$(conda shell.xonsh hook),` | str |
    | `rever/activities/push_tag.xsh` | `raw_remote = [r for r in $(git remote -v).split()` | argument |
    | `rever/activities/pypi.xsh` | `pypi_name = $($PYTHON setup.py --name).strip()` | argument |
    | `rever/activities/gcloud.xsh` | `account = $(gcloud config get-value account).strip()` | argument |
    | `rever/vcsutils.xsh` | `return $(git rev-parse --abbrev-ref HEAD).strip()` | argument |
    | `rever/vcsutils.xsh` | `return $(git rev-parse HEAD).strip()` | argument |
    | `rever/vcsutils.xsh` | `tag = $(git describe --abbrev=0 --tags)`, `return tag.strip()` | argument |
    | `rever/vcsutils.xsh` | `root = $(git rev-parse --show-toplevel)`, `return root.strip()` | argument |
    | `rever/vcsutils.xsh` | `lines = $(git log "--format=%aN<%aE>").strip().splitlines()` | arguments list |
    | `rever/vcsutils.xsh` | `for line in $(git shortlog @(args)).splitlines():` | arguments list |
    | `rever/vcsutils.xsh` | `for line in $(git shortlog @(args)).splitlines():` | arguments list |
    | `rever/vcsutils.xsh` | `for line in $(git log --encoding=utf-8 --full-history --reverse "--format=format:%ae:%at").splitlines():` | arguments list |
    | `rever/tools.xsh` | `keys = $(gpg --list-keys)` | str |
    | `rever/authors.xsh` | `log = $(git log --sparse --full-history @(fmt) @(refs))` | str |
    | `rever/docker.xsh` | `imageid = $(docker images -q @(image)).strip()` | mostly argument |
    | `rever/docker.xsh` | `out = $(docker run --help)` | str |

    </details>

2. Usage of `!()` operator is low:

    <details>

    ```
    grep -ri '!(.*)' ./rever
    ./rever/activities/ghpages.xsh:                p = !(git diff --exit-code --quiet).rtn
    ./rever/activities/ghpages.xsh:                q = !(git diff --cached --exit-code --quiet).rtn
    ./rever/activity.xsh:        p = !(which $REVER_VCS)
    ./rever/activity.xsh:            if !(which @(cmd)):
    ./rever/tools.xsh:    if not bool(!(which gpg)):
    ```

    </details>

3. Usage of `![]` is the same as `$[]` because the returning value is not used in 83% of cases:

    <details>

    ```
    grep -ri '!\[.*\]' ./rever
    ./rever/activities/ghrelease.xsh:    ![git archive -9 --format=tar.gz --prefix=@(folder_name)/ -o @(fname) @(tag)]
    ./rever/activities/ghpages.xsh:            ![git clone @(repo) @(repo_dir)]
    ./rever/activities/pypi.xsh:        p = ![$PYTHON setup.py @(build_commands)]
    ./rever/activities/pypi.xsh:            p = ![twine @(upload_args) $dist_dir/*]
    ./rever/activities/forge.xsh:            p = ![git clone @(feedstock_origin) @(feedstock_dir)]
    ./rever/activities/forge.xsh:        rerender = ![conda-smithy regenerate --check]
    ./rever/activities/gcloud.xsh:        ![gcloud auth application-default login]
    ./rever/activities/gcloud.xsh:        ![gcloud auth login]
    ./rever/activities/gcloud.xsh:            ![kubectl set image deployment/$GCLOUD_CONTAINER_NAME $GCLOUD_CONTAINER_NAME=$GCLOUD_DOCKER_HOST/$GCLOUD_DOCKER_ORG/$GCLOUD_DOCKER_REPO:$VERSION]
    ./rever/activities/gcloud.xsh:                ![kubectl rollout status deployment/$GCLOUD_CONTAINER_NAME]
    ./rever/activities/gcloud.xsh:        ![kubectl rollout undo deployment/$GCLOUD_CONTTAINER_NAME]
    ./rever/activities/docker.xsh:            ![docker build @(args)]
    ./rever/activities/docker.xsh:            ![docker push @(tag)]
    ./rever/activities/docker.xsh:            ![docker build @(args) .]
    ./rever/activities/docker.xsh:                ![docker push @(tag)]
    ./rever/vcsutils.xsh:    ![git init @(tempd)]
    ./rever/vcsutils.xsh:            ![git checkout -b __rever__]
    ./rever/vcsutils.xsh:            ![git commit --allow-empty -m 'Checking rever permissions']
    ./rever/vcsutils.xsh:            ![git push --force @(remote) __rever__:__rever__]
    ./rever/vcsutils.xsh:            ![git push --force @(remote) :__rever__]
    ./rever/tools.xsh:    ![cd @(d)]
    ./rever/tools.xsh:    ![cd @(old_d)]
    ./rever/docker.xsh:    ![docker build -t @(image) -f @(dockerfile) --no-cache .]
    ./rever/docker.xsh:    ![docker run -t @(env_args) @(mount_args) @(image) @(command)]
    ```

    </details>

As result this statistics on the code that written by xonsh maintainers and contributors completely supports the XEP-2 changes where we use: `$()` to get arguments, `$().str` shortcut to rarely used use case of getting string, finally we merged `!()` `![]` operators into `$` and removed them.
@anki-code I like your proposal, especially as remembering the meaning of all the different command substitution operators is the most difficult part of using xonsh IMO. It doesn't look like there's been any progress or discussion for a few years, is there any plan to move forward with this proposal or is something blocking it?
I have no time to implement this but I still believe in this. There are no direct blockers.
This seems like a really good proposal.
One thing that isn't adressed, the `.out` attribute would contain the raw stdout, so wouldn't it make sense to have a `.err` attribute as well for raw stderr?
Great proposal. Xonsh's current 5+ expansion operators seem redundant.

Question: Do we need a separate operator for uncaptured subprocess?

I'm new to xonsh, so I'm not familiar with all of the use cases, but at first glance it doesn't look like there is much use for an uncaptured subprocess operator.

In python mode, if you don't need to use the value of a $() subprocess result, then just don't use it and it will be removed when the scope ends. If you need to see stdout in the terminal, then `print($(...))` should work, right?

In subproc mode, there doesn't seem to be a purpose to uncaptured subprocesses at all, over just making separate statements `foo; bar`.
> Question: Do we need a separate operator for uncaptured subprocess?

The [xonsh tutorial shows](https://xon.sh/tutorial.html#uncaptured-subprocess-with-and) different use cases. In addition I would say that `![]` is needed to explicitly tells that this is subprocess command.

Resolving patch:
diff --git a/xonsh/environ.py b/xonsh/environ.py
--- a/xonsh/environ.py
+++ b/xonsh/environ.py
@@ -975,6 +975,12 @@ class GeneralSetting(Xettings):
         "not always happen.",
         is_configurable=False,
     )
+    XONSH_SUBPROC_OUTPUT_FORMAT = Var.with_default(
+        "stream_lines",
+        "Set output format for subprocess e.g. ``du $(ls)``. "
+        "By default (``stream_lines``) subprocess operator returns text output. "
+        "Set ``list_lines`` to have list of lines.",
+    )
     XONSH_CAPTURE_ALWAYS = Var.with_default(
         False,
         "Try to capture output of commands run without explicit capturing.\n"
diff --git a/xonsh/procs/pipelines.py b/xonsh/procs/pipelines.py
--- a/xonsh/procs/pipelines.py
+++ b/xonsh/procs/pipelines.py
@@ -629,15 +629,33 @@ def inp(self):
         """Creates normalized input string from args."""
         return " ".join(self.args)
 
+    def get_formatted_lines(self, lines):
+        """Format output lines."""
+        format = XSH.env.get("XONSH_SUBPROC_OUTPUT_FORMAT", "stream_lines")
+        if format == "stream_lines":
+            if len(lines) == 1:
+                return lines[0].rstrip("\n")
+            else:
+                return "".join(lines)
+        elif format == "list_lines":
+            if not lines:
+                return lines
+            elif len(lines) == 1:
+                return lines[0].rstrip("\n")
+            else:
+                return [line.rstrip("\n") for line in lines]
+        elif callable(format):
+            return format(lines)
+
     @property
     def output(self):
         """Non-blocking, lazy access to output"""
         if self.ended:
             if self._output is None:
-                self._output = "".join(self.lines)
+                self._output = self.get_formatted_lines(self.lines)
             return self._output
         else:
-            return "".join(self.lines)
+            return self.get_formatted_lines(self.lines)
 
     @property
     def out(self):
diff --git a/xonsh/procs/specs.py b/xonsh/procs/specs.py
--- a/xonsh/procs/specs.py
+++ b/xonsh/procs/specs.py
@@ -588,6 +588,7 @@ def build(kls, cmd, *, cls=subprocess.Popen, **kwargs):
         # modifications that do not alter cmds may come before creating instance
         spec = kls(cmd, cls=cls, **kwargs)
         # modifications that alter cmds must come after creating instance
+        spec.resolve_args_list()
         # perform initial redirects
         spec.resolve_redirects()
         # apply aliases
@@ -599,6 +600,13 @@ def build(kls, cmd, *, cls=subprocess.Popen, **kwargs):
         spec.resolve_stack()
         return spec
 
+    def resolve_args_list(self):
+        """Weave a list of arguments into a command."""
+        resolved_cmd = []
+        for c in self.cmd:
+            resolved_cmd += c if isinstance(c, list) else [c]
+        self.cmd = resolved_cmd
+
     def resolve_redirects(self):
         """Manages redirects."""
         new_cmd = []


Test patch:
diff --git a/tests/procs/test_specs.py b/tests/procs/test_specs.py
--- a/tests/procs/test_specs.py
+++ b/tests/procs/test_specs.py
@@ -80,7 +80,7 @@ def test_capture_always(
     exp = "HELLO\nBYE\n"
     cmds = [["echo", "-n", exp]]
     if pipe:
-        exp = exp.splitlines()[1] + "\n"  # second line
+        exp = exp.splitlines()[1]  # second line
         cmds += ["|", ["grep", "--color=never", exp.strip()]]
 
     if alias_type:
@@ -134,6 +134,28 @@ def test_capture_always(
     assert exp in capfd.readouterr().out
 
 
+@skip_if_on_windows
+@pytest.mark.parametrize(
+    "cmds, exp_stream_lines, exp_list_lines",
+    [
+        ([["echo", "-n", "1"]], "1", "1"),
+        ([["echo", "-n", "1\n"]], "1", "1"),
+        ([["echo", "-n", "1\n2\n3\n"]], "1\n2\n3\n", ["1", "2", "3"]),
+        ([["echo", "-n", "1\r\n2\r3\r\n"]], "1\n2\n3\n", ["1", "2", "3"]),
+        ([["echo", "-n", "1\n2\n3"]], "1\n2\n3", ["1", "2", "3"]),
+        ([["echo", "-n", "1\n2 3"]], "1\n2 3", ["1", "2 3"]),
+    ],
+)
+def test_subproc_output_format(cmds, exp_stream_lines, exp_list_lines, xonsh_session):
+    xonsh_session.env["XONSH_SUBPROC_OUTPUT_FORMAT"] = "stream_lines"
+    output = run_subproc(cmds, "stdout")
+    assert output == exp_stream_lines
+
+    xonsh_session.env["XONSH_SUBPROC_OUTPUT_FORMAT"] = "list_lines"
+    output = run_subproc(cmds, "stdout")
+    assert output == exp_list_lines
+
+
 @skip_if_on_windows
 @pytest.mark.parametrize(
     "captured, exp_is_none",
@@ -169,6 +191,11 @@ def __call__(self, *args, **kwargs):
     assert proc.f == obj
 
 
+def test_specs_resolve_args_list():
+    spec = cmds_to_specs([["echo", ["1", "2", "3"]]], captured="stdout")[0]
+    assert spec.cmd[-3:] == ["1", "2", "3"]
+
+
 @pytest.mark.parametrize("captured", ["hiddenobject", False])
 def test_procproxy_not_captured(xession, captured):
     xession.aliases["tst"] = lambda: 0
diff --git a/tests/test_integrations.py b/tests/test_integrations.py
--- a/tests/test_integrations.py
+++ b/tests/test_integrations.py
@@ -410,7 +410,7 @@ def _echo(args):
 
 print(check_output(["echo", "hello"]).decode("utf8"))
 """,
-        "hello\n\n",
+        "hello\n",
         0,
     ),
     #
diff --git a/tests/test_pipelines.py b/tests/test_pipelines.py
--- a/tests/test_pipelines.py
+++ b/tests/test_pipelines.py
@@ -36,14 +36,15 @@ def patched_events(monkeypatch, xonsh_events, xonsh_session):
 
 
 @pytest.mark.parametrize(
-    "cmdline, stdout, stderr",
+    "cmdline, stdout, stderr, raw_stdout",
     (
-        ("!(echo hi)", "hi\n", ""),
-        ("!(echo hi o>e)", "", "hi\n"),
+        ("!(echo hi)", "hi", "", "hi\n"),
+        ("!(echo hi o>e)", "", "hi\n", ""),
         pytest.param(
             "![echo hi]",
-            "hi\n",
+            "hi",
             "",
+            "hi\n",
             marks=pytest.mark.xfail(
                 ON_WINDOWS,
                 reason="ConsoleParallelReader doesn't work without a real console",
@@ -53,34 +54,39 @@ def patched_events(monkeypatch, xonsh_events, xonsh_session):
             "![echo hi o>e]",
             "",
             "hi\n",
+            "",
             marks=pytest.mark.xfail(
                 ON_WINDOWS, reason="stderr isn't captured in ![] on windows"
             ),
         ),
         pytest.param(
-            r"!(echo 'hi\nho')", "hi\nho\n", "", marks=skip_if_on_windows
+            r"!(echo 'hi\nho')", "hi\nho\n", "", "hi\nho\n", marks=skip_if_on_windows
         ),  # won't work with cmd
         # for some reason cmd's echo adds an extra space:
         pytest.param(
-            r"!(cmd /c 'echo hi && echo ho')", "hi \nho\n", "", marks=skip_if_on_unix
+            r"!(cmd /c 'echo hi && echo ho')",
+            "hi \nho\n",
+            "",
+            "hi \nho\n",
+            marks=skip_if_on_unix,
         ),
-        ("!(echo hi | grep h)", "hi\n", ""),
-        ("!(echo hi | grep x)", "", ""),
+        ("!(echo hi | grep h)", "hi", "", "hi\n"),
+        ("!(echo hi | grep x)", "", "", ""),
     ),
 )
-def test_command_pipeline_capture(cmdline, stdout, stderr, xonsh_execer):
+def test_command_pipeline_capture(cmdline, stdout, stderr, raw_stdout, xonsh_execer):
     pipeline: CommandPipeline = xonsh_execer.eval(cmdline)
     assert pipeline.out == stdout
     assert pipeline.err == (stderr or None)
-    assert pipeline.raw_out == stdout.replace("\n", os.linesep).encode()
+    assert pipeline.raw_out == raw_stdout.replace("\n", os.linesep).encode()
     assert pipeline.raw_err == stderr.replace("\n", os.linesep).encode()
 
 
 @pytest.mark.parametrize(
     "cmdline, output",
     (
-        ("echo hi", "hi\n"),
-        ("echo hi | grep h", "hi\n"),
+        ("echo hi", "hi"),
+        ("echo hi | grep h", "hi"),
         ("echo hi | grep x", ""),
         pytest.param("echo -n hi", "hi", marks=skip_if_on_windows),
     ),
@@ -90,7 +96,7 @@ def test_simple_capture(cmdline, output, xonsh_execer):
 
 
 def test_raw_substitution(xonsh_execer):
-    assert xonsh_execer.eval("$(echo @(b'bytes!'))") == "bytes!\n"
+    assert xonsh_execer.eval("$(echo @(b'bytes!'))") == "bytes!"
 
 
 @pytest.mark.parametrize(
@@ -102,7 +108,7 @@ def test_raw_substitution(xonsh_execer):
         ("int(!(nocommand))", 1),
         ("hash(!(echo 1))", 0),
         ("hash(!(nocommand))", 1),
-        ("str(!(echo 1))", "1\n"),
+        ("str(!(echo 1))", "1"),
         ("str(!(nocommand))", ""),
         ("!(echo 1) == 0", True),
         ("!(nocommand) == 1", True),