Repository: clap-rs/clap (language: rust)

Issue:
[modular] Pull out a clap_lex
### Please complete the following tasks

- [X] I have searched the [discussions](https://github.com/clap-rs/clap/discussions)
- [X] I have searched the existing issues

### Clap Version

master

### Describe your use case

We are looking to modular clap
- We can look for opportunities to shrink clap
- We can share logic with other argument parsers

As previously discussed at https://github.com/clap-rs/clap/discussions/2615, one part of this would be pulling out a `clap_lex` crate, similar to `lexopt`.

### Describe the solution you'd like

Clap is a wrapper around `clap_lex`

### Alternatives, if applicable

Expand `lexopt` for our needs (probably easier to do an extraction refactor)

### Additional Context

_No response_


Discussion / hints:
I agree with the author of `lexopt` in that discussion. I think there are quite a few scenarios clap is currently used for where a CFG parser would not work. (To give more context, lexer is used in CFG parsers and is useless in PEG parsers).

From all the examples of clap usage I have seen for the past 2 years, I am confident that we need to keep this as a PEG parser. We can make this PEG parser modular by implementing the plugins proposal in #2832.

I would nominate to close this issue (take your time to do research/experimentation) if needed.
> I agree with the author of lexopt in that discussion. I think there are quite a few scenarios clap is currently used for where a CFG parser would not work. (To give more context, lexer is used in CFG parsers and is useless in PEG parsers).

The `lexopt` author didn't saw it was impossible, just said `lexopt` is insufficient.  They went on to say:

> I like the idea of clap_lexer, though, so I'll follow this discussion with interest.

kbnapp also [thought it was possible](https://www.reddit.com/r/rust/comments/oley2c/lexopt_a_minimalist_pedantic_argument_parser/h5evqhk/)

> Very cool, I like the API a lot. I keep wishing I had more time on my hands to make something analogous to this as an internal clap API that all the clap features are built on top of which would allow using cargo features to aggressively trim down clap's features in an opt in/out manner (and thus be able to get that sweet small binary size).

Could you give a concrete example of why `lexopt`s general approach is incompatible with clap?

> We can make this PEG parser modular by implementing the plugins proposal in #2832.

A future experiment I want to do is something like `clap_derive` that directly populates the struct, without using the builder API.

In general, I would like to raise the bar for all of these non-clap parsers so we can have discussions beyond "does it support non-utf8?" :)

These are what I would like to see unlocked by decoupling our lowest level parsing logic from all of the clap's policy.  https://github.com/clap-rs/clap/discussions/2832 is insufficient for this.

I'm also hopeful this will help us reduce bloat, directly and indirectly by
- Giving us an opportunity to find ways to streamline logic that has grown organically over time to help us reduce our bloat.
- Have more definiable pieces that we can attribute cost towards
I think a simple example here would be `-1`. Would you parse it as a short option or a value by the lexer? There are other ambiguous things like this that would indicate that this is not a context free grammar. And lexers are only useful in CFG.
I get the feeling you are thinking we'd have a signature like
```rust
pub fn lex(args: &[OsString) -> Vec<Token>;
```

Is that correct?

If so, then maybe we aren't using terms correctly but that isn't what `lexopt` does which is the base of my suggestion (both in name and in functionality).

[`lexopt`](https://docs.rs/lexopt/0.1.0/lexopt/) processes one token at a time, driven by the caller.  A simple call to `next()` will auto-detect the token type but you can tell it the next token should be a value by calling `value()`.  We could even have a `peek_next()` to handle some of our precedence cases between values and args or values and subcommands.

This leaves it to the caller to handle what `-1` should be, along with whether `-svalue` is `-s value` or `-s -v -a -l -u -e`.
I understand what you are saying now. Let's agree to not call this `lex` anywhere because it gives off the wrong impression.

Once this is implemented, do you envision this to be used by the plugins?
> Let's agree to not call this lex anywhere because it gives off the wrong impression.

So far I've seen little confusion over `lexopt`s name and suspect it helped people get the intent.  Now, if you have an idea for a better name, I'm open.

> Once this is implemented, do you envision this to be used by the plugins?

I've not seen a case where a plugin would have access to this.  The benefits I see to this work are
- Help clean up clap, hopefully reducing our code size along the way
- Provide a solid base of up and coming argument parser to build on top of, rather than everyone reinventing the wheel, but poorly
- One day, open the door to experimenting on a `clap_derive` that doesn't use the builder.
> Provide a solid base of up and coming argument parser to build on top of, rather than everyone reinventing the wheel, but poorly

It seems you could end up with something tightly coupled to clap if it needs to know about settings like [`AppSettings::AllowNegativeNumbers`](https://docs.rs/clap/2.33.3/clap/enum.AppSettings.html#variant.AllowNegativeNumbers). And I don't know how clap implements features like pointing out that while some option is invalid *here* it would be valid if you put it *there* after the subcommand, but that also seems like it could get in the way of being generic.

Clap is large and ambitious. If some third-party generic basis already existed then I wouldn't expect clap to use it because it's working at a scale where a bespoke solution is worth the effort. It's like how rustc and GCC use handwritten parsers instead of parser generators.

A basis shared by multiple parsers could be a good idea, but I would guess that extracting it from clap isn't the way to go. (Then again, I know almost nothing about clap's internals.)
My hope is that it doesn't need to know about any settings like that, those would all be driven by the caller. The main risk is that this doesn't simplify our code but makes it more complicated. If it doesn't work then we've learned something. 
> So far I've seen little confusion over lexopts name and suspect it helped people get the intent.

> This library may also be useful if a lot of control is desired, like when the exact argument order matters or not all options are known ahead of time. It could be considered more of a lexer than a parser.

It was more of a general idea, but that library is not lexer. A better name would be `clap_parser` or `clap_parse_stream` etc..

> I've not seen a case where a plugin would have access to this.

> My hope is that it doesn't need to know about any settings like that, those would all be driven by the caller. 

And I envision that the callers here will be plugins.
> And I envision that the callers here will be plugins.

I'm not aware of this fitting in with any of our existing plugin talk (value validation currently being the most concrete).
We discussed that in https://github.com/clap-rs/clap/discussions/2832#discussioncomment-1465059.
> We discussed that in [#2832 (comment)](https://github.com/clap-rs/clap/discussions/2832#discussioncomment-1465059).

There was not enough detail on that for me to know what its referring to or how it ties into this

> > So far I've seen little confusion over lexopts name and suspect it helped people get the intent.
> 
> > This library may also be useful if a lot of control is desired, like when the exact argument order matters or not all options are known ahead of time. It could be considered more of a lexer than a parser.
> 
> It was more of a general idea, but that library is not lexer. A better name would be `clap_parser` or `clap_parse_stream` etc..

All it does is tokenization.  All of the policy of how to parse those tokens is up to the caller.  That sounds closer to a lexer in name than anything else and calling it a parser would be overselling it.  Also, the term "parser" is being used in so many places in clap in different ways, its good for us to look for more specific terms.
> All it does is tokenization.

But we are not planning to do tokenization. We are planning to let them read the items one by one without us categorizing them. That's it. And that's called a parse stream IIUC. Ex: `syn` uses the same word too.

Lexopt does actually categorize those args (and thus tokenize) but as @blyxxyz pointed out, it doesn't handle the corner cases that clap does.
At least from my searching, `syn` is the only user of that term. I did see "Token Stream" used in a similar manner.

> We are planning to let them read the items one by one without us categorizing them

I see what we create being relatively similar to `lexopt` except it will have peek functionality.
> I see what we create being relatively similar to lexopt except it will have peek functionality.

I am not sure how that would work with those corner cases (ex: negative values, multiple values) since we definitely need to leave up the categorization to individual parsers. But I guess it's a design for later.

I realise that we are just bikeshedding on the name of this module at this point. But yeah, I felt that the term `lex` would be too confusing since it's not CFG.

Fix patch:
diff --git a/Cargo.toml b/Cargo.toml
index b05e55fd1aa..c55134d7cda 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,7 @@
 [workspace]
 members = [
 	"clap_derive",
+	"clap_lex",
 	"clap_complete",
 	"clap_complete_fig",
 	"clap_mangen",
@@ -118,11 +119,11 @@ path = "benches/06_rustup.rs"
 
 [dependencies]
 clap_derive = { path = "./clap_derive", version = "3.1.7", optional = true }
+clap_lex = { path = "./clap_lex", version = "0.1.0" }
 bitflags = "1.2"
 textwrap = { version = "0.15.0", default-features = false, features = [] }
 unicase = { version = "2.6", optional = true }
 indexmap = "1.0"
-os_str_bytes = "6.0"
 strsim = { version = "0.10",  optional = true }
 yaml-rust = { version = "0.4.1",  optional = true }
 atty = { version = "0.2",  optional = true }
diff --git a/README.md b/README.md
index a1d750348a0..8e8fd3b3283 100644
--- a/README.md
+++ b/README.md
@@ -120,6 +120,7 @@ Why use the procedural [Builder API](https://github.com/clap-rs/clap/blob/v3.1.8
 - [wild](https://crates.io/crates/wild) for supporting wildcards (`*`) on Windows like you do Linux
 - [argfile](https://crates.io/crates/argfile) for loading additional arguments from a file (aka response files)
 - [shadow-rs](https://crates.io/crates/shadow-rs) for generating `Command::long_version`
+- [clap_lex](https://crates.io/crates/clap_lex) for a lighter-weight, battle-tested CLI parser
 - [clap_mangen](https://crates.io/crates/clap_mangen) for generating man page source (roff)
 - [clap_complete](https://crates.io/crates/clap_complete) for shell completion support
 - [clap-verbosity-flag](https://crates.io/crates/clap-verbosity-flag)
diff --git a/clap_lex/CHANGELOG.md b/clap_lex/CHANGELOG.md
new file mode 100644
index 00000000000..16a17c25aaa
--- /dev/null
+++ b/clap_lex/CHANGELOG.md
@@ -0,0 +1,11 @@
+# Change Log
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/)
+and this project adheres to [Semantic Versioning](http://semver.org/).
+
+<!-- next-header -->
+## [Unreleased] - ReleaseDate
+
+<!-- next-url -->
+[Unreleased]: https://github.com/clap-rs/clap/compare/ce71b08a3fe28c640dc6e17f6f5bb1452bd6d6d8...HEAD
diff --git a/clap_lex/CONTRIBUTING.md b/clap_lex/CONTRIBUTING.md
new file mode 100644
index 00000000000..6d24d2f6f32
--- /dev/null
+++ b/clap_lex/CONTRIBUTING.md
@@ -0,0 +1,3 @@
+# How to Contribute
+
+See the [clap-wide CONTRIBUTING.md](../CONTRIBUTING.md).  This will contain `clap_lex` specific notes.
diff --git a/clap_lex/Cargo.toml b/clap_lex/Cargo.toml
new file mode 100644
index 00000000000..2c82529b98e
--- /dev/null
+++ b/clap_lex/Cargo.toml
@@ -0,0 +1,39 @@
+[package]
+name = "clap_lex"
+version = "0.1.0"
+edition = "2018"
+include = [
+	"src/**/*",
+	"Cargo.toml",
+	"LICENSE-*",
+	"README.md"
+]
+description = "Minimal, flexible command line parser"
+repository = "https://github.com/clap-rs/clap/tree/master/clap_lex"
+documentation = "https://docs.rs/clap_lex"
+keywords = [
+	"argument",
+	"cli",
+	"arg",
+	"parser",
+	"parse"
+]
+categories = ["command-line-interface"]
+license = "MIT OR Apache-2.0"
+readme = "README.md"
+
+[package.metadata.release]
+pre-release-replacements = [
+  {file="CHANGELOG.md", search="Unreleased", replace="{{version}}", min=1},
+  {file="CHANGELOG.md", search="\\.\\.\\.HEAD", replace="...{{tag_name}}", exactly=1},
+  {file="CHANGELOG.md", search="ReleaseDate", replace="{{date}}", min=1},
+  {file="CHANGELOG.md", search="<!-- next-header -->", replace="<!-- next-header -->\n## [Unreleased] - ReleaseDate\n", exactly=1},
+  {file="CHANGELOG.md", search="<!-- next-url -->", replace="<!-- next-url -->\n[Unreleased]: https://github.com/clap-rs/clap/compare/{{tag_name}}...HEAD", exactly=1},
+  {file="README.md", search="github.com/clap-rs/clap/blob/[^/]+/", replace="github.com/clap-rs/clap/blob/{{tag_name}}/", exactly=4, prerelease = true},
+]
+
+[lib]
+bench = false
+
+[dependencies]
+os_str_bytes = "6.0"
diff --git a/clap_lex/LICENSE-APACHE b/clap_lex/LICENSE-APACHE
new file mode 100644
index 00000000000..261eeb9e9f8
--- /dev/null
+++ b/clap_lex/LICENSE-APACHE
@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
diff --git a/clap_lex/LICENSE-MIT b/clap_lex/LICENSE-MIT
new file mode 100644
index 00000000000..5acedf04122
--- /dev/null
+++ b/clap_lex/LICENSE-MIT
@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015-2016 Kevin B. Knapp
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/clap_lex/README.md b/clap_lex/README.md
new file mode 100644
index 00000000000..ceff62127e6
--- /dev/null
+++ b/clap_lex/README.md
@@ -0,0 +1,19 @@
+<!-- omit in TOC -->
+# clap_lex
+
+> **Minimal, flexible command line parser**
+
+[![Crates.io](https://img.shields.io/crates/v/clap_lex?style=flat-square)](https://crates.io/crates/clap_lex)
+[![Crates.io](https://img.shields.io/crates/d/clap_lex?style=flat-square)](https://crates.io/crates/clap_lex)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square)](https://github.com/clap-rs/clap/blob/clap_lex-v3.1.1/LICENSE-APACHE)
+[![License](https://img.shields.io/badge/license-MIT-blue?style=flat-square)](https://github.com/clap-rs/clap/blob/clap_lex-v3.1.1/LICENSE-MIT)
+
+Dual-licensed under [Apache 2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT).
+
+1. [About](#about)
+2. [API Reference](https://docs.rs/clap_lex)
+3. [Questions & Discussions](https://github.com/clap-rs/clap/discussions)
+4. [CONTRIBUTING](https://github.com/clap-rs/clap/blob/clap_lex-v3.1.1/clap_lex/CONTRIBUTING.md)
+5. [Sponsors](https://github.com/clap-rs/clap/blob/clap_lex-v3.1.1/README.md#sponsors)
+
+## About
diff --git a/clap_lex/src/lib.rs b/clap_lex/src/lib.rs
new file mode 100644
index 00000000000..4d9878aa35d
--- /dev/null
+++ b/clap_lex/src/lib.rs
@@ -0,0 +1,472 @@
+//! Minimal, flexible command-line parser
+//!
+//! As opposed to a declarative parser, this processes arguments as a stream of tokens.  As lexing
+//! a command-line is not context-free, we rely on the caller to decide how to interpret the
+//! arguments.
+//!
+//! # Examples
+//!
+//! ```rust
+//! # use std::path::PathBuf;
+//! # type BoxedError = Box<dyn std::error::Error + Send + Sync>;
+//! #[derive(Debug)]
+//! struct Args {
+//!     paths: Vec<PathBuf>,
+//!     color: Color,
+//!     verbosity: usize,
+//! }
+//!
+//! #[derive(Debug)]
+//! enum Color {
+//!     Always,
+//!     Auto,
+//!     Never,
+//! }
+//!
+//! impl Color {
+//!     fn parse(s: Option<&clap_lex::RawOsStr>) -> Result<Self, BoxedError> {
+//!         let s = s.map(|s| s.to_str().ok_or(s));
+//!         match s {
+//!             Some(Ok("always")) | Some(Ok("")) | None => {
+//!                 Ok(Color::Always)
+//!             }
+//!             Some(Ok("auto")) => {
+//!                 Ok(Color::Auto)
+//!             }
+//!             Some(Ok("never")) => {
+//!                 Ok(Color::Never)
+//!             }
+//!             Some(invalid) => {
+//!                 Err(format!("Invalid value for `--color`, {:?}", invalid).into())
+//!             }
+//!         }
+//!     }
+//! }
+//!
+//! fn parse_args(
+//!     raw: impl IntoIterator<Item=impl Into<std::ffi::OsString>>
+//! ) -> Result<Args, BoxedError> {
+//!     let mut args = Args {
+//!         paths: Vec::new(),
+//!         color: Color::Auto,
+//!         verbosity: 0,
+//!     };
+//!
+//!     let raw = clap_lex::RawArgs::new(raw);
+//!     let mut cursor = raw.cursor();
+//!     raw.next(&mut cursor);  // Skip the bin
+//!     while let Some(arg) = raw.next(&mut cursor) {
+//!         if arg.is_escape() {
+//!             args.paths.extend(raw.remaining(&mut cursor).map(PathBuf::from));
+//!         } else if arg.is_stdio() {
+//!             args.paths.push(PathBuf::from("-"));
+//!         } else if let Some((long, value)) = arg.to_long() {
+//!             match long {
+//!                 Ok("verbose") => {
+//!                     if let Some(value) = value {
+//!                         return Err(format!("`--verbose` does not take a value, got `{:?}`", value).into());
+//!                     }
+//!                     args.verbosity += 1;
+//!                 }
+//!                 Ok("color") => {
+//!                     args.color = Color::parse(value)?;
+//!                 }
+//!                 _ => {
+//!                     return Err(
+//!                         format!("Unexpected flag: --{}", arg.display()).into()
+//!                     );
+//!                 }
+//!             }
+//!         } else if let Some(mut shorts) = arg.to_short() {
+//!             while let Some(short) = shorts.next_flag() {
+//!                 match short {
+//!                     Ok('v') => {
+//!                         args.verbosity += 1;
+//!                     }
+//!                     Ok('c') => {
+//!                         let value = shorts.next_value_os();
+//!                         args.color = Color::parse(value)?;
+//!                     }
+//!                     Ok(c) => {
+//!                         return Err(format!("Unexpected flag: -{}", c).into());
+//!                     }
+//!                     Err(e) => {
+//!                         return Err(format!("Unexpected flag: -{}", e.to_str_lossy()).into());
+//!                     }
+//!                 }
+//!             }
+//!         } else {
+//!             args.paths.push(PathBuf::from(arg.to_value_os().to_os_str().into_owned()));
+//!         }
+//!     }
+//!
+//!     Ok(args)
+//! }
+//!
+//! let args = parse_args(["bin", "--hello", "world"]);
+//! println!("{:?}", args);
+//! ```
+
+use std::ffi::OsStr;
+use std::ffi::OsString;
+
+pub use std::io::SeekFrom;
+
+pub use os_str_bytes::RawOsStr;
+pub use os_str_bytes::RawOsString;
+
+/// Command-line arguments
+#[derive(Default, Clone, Debug, PartialEq, Eq)]
+pub struct RawArgs {
+    items: Vec<OsString>,
+}
+
+impl RawArgs {
+    //// Create an argument list to parse
+    ///
+    /// **NOTE:** The argument returned will be the current binary.
+    ///
+    /// # Example
+    ///
+    /// ```rust,no_run
+    /// # use std::path::PathBuf;
+    /// let raw = clap_lex::RawArgs::from_args();
+    /// let mut cursor = raw.cursor();
+    /// let _bin = raw.next_os(&mut cursor);
+    ///
+    /// let mut paths = raw.remaining(&mut cursor).map(PathBuf::from).collect::<Vec<_>>();
+    /// println!("{:?}", paths);
+    /// ```
+    pub fn from_args() -> Self {
+        Self::new(std::env::args_os())
+    }
+
+    //// Create an argument list to parse
+    ///
+    /// # Example
+    ///
+    /// ```rust,no_run
+    /// # use std::path::PathBuf;
+    /// let raw = clap_lex::RawArgs::new(["bin", "foo.txt"]);
+    /// let mut cursor = raw.cursor();
+    /// let _bin = raw.next_os(&mut cursor);
+    ///
+    /// let mut paths = raw.remaining(&mut cursor).map(PathBuf::from).collect::<Vec<_>>();
+    /// println!("{:?}", paths);
+    /// ```
+    pub fn new(iter: impl IntoIterator<Item = impl Into<std::ffi::OsString>>) -> Self {
+        let iter = iter.into_iter();
+        Self::from(iter)
+    }
+
+    /// Create a cursor for walking the arguments
+    ///
+    /// # Example
+    ///
+    /// ```rust,no_run
+    /// # use std::path::PathBuf;
+    /// let raw = clap_lex::RawArgs::new(["bin", "foo.txt"]);
+    /// let mut cursor = raw.cursor();
+    /// let _bin = raw.next_os(&mut cursor);
+    ///
+    /// let mut paths = raw.remaining(&mut cursor).map(PathBuf::from).collect::<Vec<_>>();
+    /// println!("{:?}", paths);
+    /// ```
+    pub fn cursor(&self) -> ArgCursor {
+        ArgCursor::new()
+    }
+
+    /// Advance the cursor, returning the next [`ParsedArg`]
+    pub fn next(&self, cursor: &mut ArgCursor) -> Option<ParsedArg<'_>> {
+        self.next_os(cursor).map(ParsedArg::new)
+    }
+
+    /// Advance the cursor, returning a raw argument value.
+    pub fn next_os(&self, cursor: &mut ArgCursor) -> Option<&OsStr> {
+        let next = self.items.get(cursor.cursor).map(|s| s.as_os_str());
+        cursor.cursor = cursor.cursor.saturating_add(1);
+        next
+    }
+
+    /// Return the next [`ParsedArg`]
+    pub fn peek(&self, cursor: &ArgCursor) -> Option<ParsedArg<'_>> {
+        self.peek_os(cursor).map(ParsedArg::new)
+    }
+
+    /// Return a raw argument value.
+    pub fn peek_os(&self, cursor: &ArgCursor) -> Option<&OsStr> {
+        self.items.get(cursor.cursor).map(|s| s.as_os_str())
+    }
+
+    /// Return all remaining raw arguments, advancing the cursor to the end
+    ///
+    /// # Example
+    ///
+    /// ```rust,no_run
+    /// # use std::path::PathBuf;
+    /// let raw = clap_lex::RawArgs::new(["bin", "foo.txt"]);
+    /// let mut cursor = raw.cursor();
+    /// let _bin = raw.next_os(&mut cursor);
+    ///
+    /// let mut paths = raw.remaining(&mut cursor).map(PathBuf::from).collect::<Vec<_>>();
+    /// println!("{:?}", paths);
+    /// ```
+    pub fn remaining(&self, cursor: &mut ArgCursor) -> impl Iterator<Item = &OsStr> {
+        let remaining = self.items[cursor.cursor..].iter().map(|s| s.as_os_str());
+        cursor.cursor = self.items.len();
+        remaining
+    }
+
+    /// Adjust the cursor's position
+    pub fn seek(&self, cursor: &mut ArgCursor, pos: SeekFrom) {
+        let pos = match pos {
+            SeekFrom::Start(pos) => pos,
+            SeekFrom::End(pos) => (self.items.len() as i64).saturating_add(pos).max(0) as u64,
+            SeekFrom::Current(pos) => (cursor.cursor as i64).saturating_add(pos).max(0) as u64,
+        };
+        let pos = (pos as usize).min(self.items.len());
+        cursor.cursor = pos;
+    }
+
+    /// Inject arguments before the [`RawArgs::next`]
+    pub fn insert(&mut self, cursor: &ArgCursor, insert_items: &[&str]) {
+        self.items.splice(
+            cursor.cursor..cursor.cursor,
+            insert_items.iter().map(OsString::from),
+        );
+    }
+}
+
+impl<I, T> From<I> for RawArgs
+where
+    I: Iterator<Item = T>,
+    T: Into<OsString>,
+{
+    fn from(val: I) -> Self {
+        Self {
+            items: val.map(|x| x.into()).collect(),
+        }
+    }
+}
+
+/// Position within [`RawArgs`]
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
+pub struct ArgCursor {
+    cursor: usize,
+}
+
+impl ArgCursor {
+    fn new() -> Self {
+        Self { cursor: 0 }
+    }
+}
+
+/// Command-line Argument
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub struct ParsedArg<'s> {
+    inner: std::borrow::Cow<'s, RawOsStr>,
+    utf8: Option<&'s str>,
+}
+
+impl<'s> ParsedArg<'s> {
+    fn new(inner: &'s OsStr) -> Self {
+        let utf8 = inner.to_str();
+        let inner = RawOsStr::new(inner);
+        Self { inner, utf8 }
+    }
+
+    /// Does the argument look like a stdio argument (`-`)
+    pub fn is_stdio(&self) -> bool {
+        self.inner.as_ref() == "-"
+    }
+
+    /// Does the argument look like an argument escape (`--`)
+    pub fn is_escape(&self) -> bool {
+        self.inner.as_ref() == "--"
+    }
+
+    /// Does the argument look like a number
+    pub fn is_number(&self) -> bool {
+        self.to_value()
+            .map(|s| s.parse::<f64>().is_ok())
+            .unwrap_or_default()
+    }
+
+    /// Treat as a long-flag
+    ///
+    /// **NOTE:** May return an empty flag.  Check [`ParsedArg::is_escape`] to separately detect `--`.
+    ///
+    /// **NOTE:** Will not match [`ParsedArg::is_stdio`], completion engines will need to check
+    /// that case.
+    pub fn to_long(&self) -> Option<(Result<&str, &RawOsStr>, Option<&RawOsStr>)> {
+        if let Some(raw) = self.utf8 {
+            let remainder = raw.strip_prefix("--")?;
+            let (flag, value) = if let Some((p0, p1)) = remainder.split_once('=') {
+                (p0, Some(p1))
+            } else {
+                (remainder, None)
+            };
+            let flag = Ok(flag);
+            let value = value.map(RawOsStr::from_str);
+            Some((flag, value))
+        } else {
+            let raw = self.inner.as_ref();
+            let remainder = raw.strip_prefix("--")?;
+            let (flag, value) = if let Some((p0, p1)) = remainder.split_once('=') {
+                (p0, Some(p1))
+            } else {
+                (remainder, None)
+            };
+            let flag = flag.to_str().ok_or(flag);
+            Some((flag, value))
+        }
+    }
+
+    /// Can treat as a long-flag
+    ///
+    /// **NOTE:** May return an empty flag.  Check [`ParsedArg::is_escape`] to separately detect `--`.
+    pub fn is_long(&self) -> bool {
+        self.inner.as_ref().starts_with("--")
+    }
+
+    /// Treat as a short-flag
+    ///
+    /// **NOTE:** Maybe return an empty flag.  Check [`ParsedArg::is_stdio`] to separately detect
+    /// `-`.
+    pub fn to_short(&self) -> Option<ShortFlags<'_>> {
+        if let Some(remainder_os) = self.inner.as_ref().strip_prefix('-') {
+            if remainder_os.starts_with('-') {
+                None
+            } else {
+                let remainder = self.utf8.map(|s| &s[1..]);
+                Some(ShortFlags::new(remainder_os, remainder))
+            }
+        } else {
+            None
+        }
+    }
+
+    /// Can treat as a short-flag
+    ///
+    /// **NOTE:** Maybe return an empty flag.  Check [`ParsedArg::is_stdio`] to separately detect
+    /// `-`.
+    pub fn is_short(&self) -> bool {
+        self.inner.as_ref().starts_with('-') && !self.is_long()
+    }
+
+    /// Treat as a value
+    ///
+    /// **NOTE:** May return a flag or an escape.
+    pub fn to_value_os(&self) -> &RawOsStr {
+        self.inner.as_ref()
+    }
+
+    /// Treat as a value
+    ///
+    /// **NOTE:** May return a flag or an escape.
+    pub fn to_value(&self) -> Result<&str, &RawOsStr> {
+        self.utf8.ok_or_else(|| self.inner.as_ref())
+    }
+
+    /// Safely print an argument that may contain non-UTF8 content
+    ///
+    /// This may perform lossy conversion, depending on the platform. If you would like an implementation which escapes the path please use Debug instead.
+    pub fn display(&self) -> impl std::fmt::Display + '_ {
+        self.inner.to_str_lossy()
+    }
+}
+
+/// Walk through short flags within a [`ParsedArg`]
+#[derive(Clone, Debug)]
+pub struct ShortFlags<'s> {
+    inner: &'s RawOsStr,
+    utf8_prefix: std::str::CharIndices<'s>,
+    invalid_suffix: Option<&'s RawOsStr>,
+}
+
+impl<'s> ShortFlags<'s> {
+    fn new(inner: &'s RawOsStr, utf8: Option<&'s str>) -> Self {
+        let (utf8_prefix, invalid_suffix) = if let Some(utf8) = utf8 {
+            (utf8, None)
+        } else {
+            split_nonutf8_once(inner)
+        };
+        let utf8_prefix = utf8_prefix.char_indices();
+        Self {
+            inner,
+            utf8_prefix,
+            invalid_suffix,
+        }
+    }
+
+    /// Move the iterator forward by `n` short flags
+    pub fn advance_by(&mut self, n: usize) -> Result<(), usize> {
+        for i in 0..n {
+            self.next().ok_or(i)?.map_err(|_| i)?;
+        }
+        Ok(())
+    }
+
+    /// No short flags left
+    pub fn is_empty(&self) -> bool {
+        self.invalid_suffix.is_none() && self.utf8_prefix.as_str().is_empty()
+    }
+
+    /// Does the short flag look like a number
+    ///
+    /// Ideally call this before doing any iterator
+    pub fn is_number(&self) -> bool {
+        self.invalid_suffix.is_none() && self.utf8_prefix.as_str().parse::<f64>().is_ok()
+    }
+
+    /// Advance the iterator, returning the next short flag on success
+    ///
+    /// On error, returns the invalid-UTF8 value
+    pub fn next_flag(&mut self) -> Option<Result<char, &'s RawOsStr>> {
+        if let Some((_, flag)) = self.utf8_prefix.next() {
+            return Some(Ok(flag));
+        }
+
+        if let Some(suffix) = self.invalid_suffix {
+            self.invalid_suffix = None;
+            return Some(Err(suffix));
+        }
+
+        None
+    }
+
+    /// Advance the iterator, returning everything left as a value
+    pub fn next_value_os(&mut self) -> Option<&'s RawOsStr> {
+        if let Some((index, _)) = self.utf8_prefix.next() {
+            self.utf8_prefix = "".char_indices();
+            self.invalid_suffix = None;
+            return Some(&self.inner[index..]);
+        }
+
+        if let Some(suffix) = self.invalid_suffix {
+            self.invalid_suffix = None;
+            return Some(suffix);
+        }
+
+        None
+    }
+}
+
+impl<'s> Iterator for ShortFlags<'s> {
+    type Item = Result<char, &'s RawOsStr>;
+
+    fn next(&mut self) -> Option<Self::Item> {
+        self.next_flag()
+    }
+}
+
+fn split_nonutf8_once(b: &RawOsStr) -> (&str, Option<&RawOsStr>) {
+    match std::str::from_utf8(b.as_raw_bytes()) {
+        Ok(s) => (s, None),
+        Err(err) => {
+            let (valid, after_valid) = b.split_at(err.valid_up_to());
+            let valid = std::str::from_utf8(valid.as_raw_bytes()).unwrap();
+            (valid, Some(after_valid))
+        }
+    }
+}
diff --git a/src/build/command.rs b/src/build/command.rs
index 03e4e00c910..bf718b4cfc5 100644
--- a/src/build/command.rs
+++ b/src/build/command.rs
@@ -3,7 +3,6 @@
 // Std
 use std::collections::HashMap;
 use std::env;
-use std::ffi::OsStr;
 use std::ffi::OsString;
 use std::fmt;
 use std::io;
@@ -11,7 +10,6 @@ use std::ops::Index;
 use std::path::Path;
 
 // Third Party
-use os_str_bytes::RawOsStr;
 #[cfg(feature = "yaml")]
 use yaml_rust::Yaml;
 
@@ -23,7 +21,7 @@ use crate::error::ErrorKind;
 use crate::error::Result as ClapResult;
 use crate::mkeymap::MKeyMap;
 use crate::output::{fmt::Colorizer, Help, HelpWriter, Usage};
-use crate::parse::{ArgMatcher, ArgMatches, Input, Parser};
+use crate::parse::{ArgMatcher, ArgMatches, Parser};
 use crate::util::ChildGraph;
 use crate::util::{color::ColorChoice, Id, Key};
 use crate::{Error, INTERNAL_ERROR_MSG};
@@ -101,7 +99,7 @@ pub struct App<'help> {
     g_settings: AppFlags,
     args: MKeyMap<'help>,
     subcommands: Vec<App<'help>>,
-    replacers: HashMap<&'help OsStr, &'help [&'help str]>,
+    replacers: HashMap<&'help str, &'help [&'help str]>,
     groups: Vec<ArgGroup<'help>>,
     current_help_heading: Option<&'help str>,
     current_disp_ord: Option<usize>,
@@ -633,11 +631,12 @@ impl<'help> App<'help> {
         I: IntoIterator<Item = T>,
         T: Into<OsString> + Clone,
     {
-        let mut it = Input::from(itr.into_iter());
+        let mut raw_args = clap_lex::RawArgs::new(itr.into_iter());
+        let mut cursor = raw_args.cursor();
 
         #[cfg(feature = "unstable-multicall")]
         if self.settings.is_set(AppSettings::Multicall) {
-            if let Some((argv0, _)) = it.next() {
+            if let Some(argv0) = raw_args.next_os(&mut cursor) {
                 let argv0 = Path::new(&argv0);
                 if let Some(command) = argv0.file_stem().and_then(|f| f.to_str()) {
                     // Stop borrowing command so we can get another mut ref to it.
@@ -648,11 +647,11 @@ impl<'help> App<'help> {
                     );
 
                     debug!("App::try_get_matches_from_mut: Reinserting command into arguments so subcommand parser matches it");
-                    it.insert(&[&command]);
+                    raw_args.insert(&cursor, &[&command]);
                     debug!("App::try_get_matches_from_mut: Clearing name and bin_name so that displayed command name starts with applet name");
                     self.name.clear();
                     self.bin_name = None;
-                    return self._do_parse(&mut it);
+                    return self._do_parse(&mut raw_args, cursor);
                 }
             }
         };
@@ -665,7 +664,7 @@ impl<'help> App<'help> {
         // to display
         // the full path when displaying help messages and such
         if !self.settings.is_set(AppSettings::NoBinaryName) {
-            if let Some((name, _)) = it.next() {
+            if let Some(name) = raw_args.next_os(&mut cursor) {
                 let p = Path::new(name);
 
                 if let Some(f) = p.file_name() {
@@ -678,7 +677,7 @@ impl<'help> App<'help> {
             }
         }
 
-        self._do_parse(&mut it)
+        self._do_parse(&mut raw_args, cursor)
     }
 
     /// Prints the short help message (`-h`) to [`io::stdout()`].
@@ -1944,7 +1943,7 @@ impl<'help> App<'help> {
     #[cfg(feature = "unstable-replace")]
     #[must_use]
     pub fn replace(mut self, name: &'help str, target: &'help [&'help str]) -> Self {
-        self.replacers.insert(OsStr::new(name), target);
+        self.replacers.insert(name, target);
         self
     }
 
@@ -3932,7 +3931,7 @@ impl<'help> App<'help> {
         self.max_w
     }
 
-    pub(crate) fn get_replacement(&self, key: &OsStr) -> Option<&[&str]> {
+    pub(crate) fn get_replacement(&self, key: &str) -> Option<&[&str]> {
         self.replacers.get(key).copied()
     }
 
@@ -3954,7 +3953,11 @@ impl<'help> App<'help> {
         }
     }
 
-    fn _do_parse(&mut self, it: &mut Input) -> ClapResult<ArgMatches> {
+    fn _do_parse(
+        &mut self,
+        raw_args: &mut clap_lex::RawArgs,
+        args_cursor: clap_lex::ArgCursor,
+    ) -> ClapResult<ArgMatches> {
         debug!("App::_do_parse");
 
         // If there are global arguments, or settings we need to propagate them down to subcommands
@@ -3965,7 +3968,7 @@ impl<'help> App<'help> {
 
         // do the real parsing
         let mut parser = Parser::new(self);
-        if let Err(error) = parser.get_matches_with(&mut matcher, it) {
+        if let Err(error) = parser.get_matches_with(&mut matcher, raw_args, args_cursor) {
             if self.is_set(AppSettings::IgnoreErrors) {
                 debug!("App::_do_parse: ignoring error: {}", error);
             } else {
@@ -4651,7 +4654,7 @@ impl<'help> App<'help> {
     }
 
     /// Find a flag subcommand name by long flag or an alias
-    pub(crate) fn find_long_subcmd(&self, long: &RawOsStr) -> Option<&str> {
+    pub(crate) fn find_long_subcmd(&self, long: &str) -> Option<&str> {
         self.get_subcommands()
             .find(|sc| sc.long_flag_aliases_to(long))
             .map(|sc| sc.get_name())
diff --git a/src/build/debug_asserts.rs b/src/build/debug_asserts.rs
index 5e6baf66644..3975fb98f9c 100644
--- a/src/build/debug_asserts.rs
+++ b/src/build/debug_asserts.rs
@@ -1,6 +1,6 @@
 use std::cmp::Ordering;
 
-use os_str_bytes::RawOsStr;
+use clap_lex::RawOsStr;
 
 use crate::build::arg::ArgProvider;
 use crate::mkeymap::KeyType;
diff --git a/src/mkeymap.rs b/src/mkeymap.rs
index 54d29e814c7..8ba3dcb3061 100644
--- a/src/mkeymap.rs
+++ b/src/mkeymap.rs
@@ -49,6 +49,15 @@ impl PartialEq<&str> for KeyType {
     }
 }
 
+impl PartialEq<str> for KeyType {
+    fn eq(&self, rhs: &str) -> bool {
+        match self {
+            KeyType::Long(l) => l == rhs,
+            _ => false,
+        }
+    }
+}
+
 impl PartialEq<OsStr> for KeyType {
     fn eq(&self, rhs: &OsStr) -> bool {
         match self {
diff --git a/src/parse/mod.rs b/src/parse/mod.rs
index 8bc348608a9..298862e1346 100644
--- a/src/parse/mod.rs
+++ b/src/parse/mod.rs
@@ -1,13 +1,13 @@
-pub mod features;
-
 mod arg_matcher;
-pub mod matches;
 mod parser;
 mod validator;
 
+pub mod features;
+pub mod matches;
+
 pub(crate) use self::arg_matcher::ArgMatcher;
 pub(crate) use self::matches::{MatchedArg, SubCommand};
-pub(crate) use self::parser::{Input, ParseState, Parser};
+pub(crate) use self::parser::{ParseState, Parser};
 pub(crate) use self::validator::Validator;
 
 pub use self::matches::{ArgMatches, Indices, OsValues, ValueSource, Values};
diff --git a/src/parse/parser.rs b/src/parse/parser.rs
index 0b515b51a53..d778b84514c 100644
--- a/src/parse/parser.rs
+++ b/src/parse/parser.rs
@@ -5,7 +5,7 @@ use std::{
 };
 
 // Third Party
-use os_str_bytes::RawOsStr;
+use clap_lex::RawOsStr;
 
 // Internal
 use crate::build::{Arg, Command};
@@ -63,7 +63,8 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
     pub(crate) fn get_matches_with(
         &mut self,
         matcher: &mut ArgMatcher,
-        it: &mut Input,
+        raw_args: &mut clap_lex::RawArgs,
+        mut args_cursor: clap_lex::ArgCursor,
     ) -> ClapResult<()> {
         debug!("Parser::get_matches_with");
         // Verify all positional assertions pass
@@ -88,22 +89,25 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         // If any arg sets .last(true)
         let contains_last = self.cmd.get_arguments().any(|x| x.is_last_set());
 
-        while let Some((arg_os, remaining_args)) = it.next() {
+        while let Some(arg_os) = raw_args.next(&mut args_cursor) {
             // Recover the replaced items if any.
-            if let Some(replaced_items) = self.cmd.get_replacement(arg_os) {
+            if let Some(replaced_items) = arg_os
+                .to_value()
+                .ok()
+                .and_then(|a| self.cmd.get_replacement(a))
+            {
                 debug!(
                     "Parser::get_matches_with: found replacer: {:?}, target: {:?}",
                     arg_os, replaced_items
                 );
-                it.insert(replaced_items);
+                raw_args.insert(&args_cursor, replaced_items);
                 continue;
             }
 
-            let arg_os = RawOsStr::new(arg_os);
             debug!(
                 "Parser::get_matches_with: Begin parsing '{:?}' ({:?})",
-                arg_os,
-                arg_os.as_raw_bytes()
+                arg_os.to_value_os(),
+                arg_os.to_value_os().as_raw_bytes()
             );
 
             // Correct pos_counter.
@@ -137,7 +141,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 );
 
                 if low_index_mults || missing_pos {
-                    let skip_current = if let Some(n) = remaining_args.get(0) {
+                    let skip_current = if let Some(n) = raw_args.peek(&args_cursor) {
                         if let Some(p) = self
                             .cmd
                             .get_positionals()
@@ -148,9 +152,10 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                             // pos_counter(which means current value cannot be a
                             // positional argument with a value next to it), assume
                             // current value matches the next arg.
-                            let n = RawOsStr::new(n);
                             self.is_new_arg(&n, p)
-                                || self.possible_subcommand(&n, valid_arg_found).is_some()
+                                || self
+                                    .possible_subcommand(n.to_value(), valid_arg_found)
+                                    .is_some()
                         } else {
                             true
                         }
@@ -182,24 +187,25 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                     || !matches!(parse_state, ParseState::Opt(_) | ParseState::Pos(_))
                 {
                     // Does the arg match a subcommand name, or any of its aliases (if defined)
-                    let sc_name = self.possible_subcommand(&arg_os, valid_arg_found);
+                    let sc_name = self.possible_subcommand(arg_os.to_value(), valid_arg_found);
                     debug!("Parser::get_matches_with: sc={:?}", sc_name);
                     if let Some(sc_name) = sc_name {
                         if sc_name == "help"
                             && !self.is_set(AS::NoAutoHelp)
                             && !self.cmd.is_disable_help_subcommand_set()
                         {
-                            self.parse_help_subcommand(remaining_args)?;
+                            self.parse_help_subcommand(raw_args.remaining(&mut args_cursor))?;
                         }
                         subcmd_name = Some(sc_name.to_owned());
                         break;
                     }
                 }
 
-                if let Some(long_arg) = arg_os.strip_prefix("--") {
+                if let Some((long_arg, long_value)) = arg_os.to_long() {
                     let parse_result = self.parse_long_arg(
                         matcher,
                         long_arg,
+                        long_value,
                         &parse_state,
                         &mut valid_arg_found,
                         trailing_values,
@@ -238,8 +244,8 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                             ));
                         }
                         ParseResult::NoMatchingArg { arg } => {
-                            let remaining_args: Vec<_> = remaining_args
-                                .iter()
+                            let remaining_args: Vec<_> = raw_args
+                                .remaining(&mut args_cursor)
                                 .map(|x| x.to_str().expect(INVALID_UTF8))
                                 .collect();
                             return Err(self.did_you_mean_error(&arg, matcher, &remaining_args));
@@ -265,7 +271,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                             unreachable!()
                         }
                     }
-                } else if let Some(short_arg) = arg_os.strip_prefix("-") {
+                } else if let Some(short_arg) = arg_os.to_short() {
                     // Arg looks like a short flag, and not a possible number
 
                     // Try to parse short args like normal, if allow_hyphen_values or
@@ -302,7 +308,8 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                             keep_state = self
                                 .flag_subcmd_at
                                 .map(|at| {
-                                    it.cursor -= 1;
+                                    raw_args
+                                        .seek(&mut args_cursor, clap_lex::SeekFrom::Current(-1));
                                     // Since we are now saving the current state, the number of flags to skip during state recovery should
                                     // be the current index (`cur_idx`) minus ONE UNIT TO THE LEFT of the starting position.
                                     self.flag_subcmd_skip = self.cur_idx.get() - at + 1;
@@ -354,7 +361,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                     // get the option so we can check the settings
                     let parse_result = self.add_val_to_arg(
                         &self.cmd[id],
-                        &arg_os,
+                        arg_os.to_value_os(),
                         matcher,
                         ValueSource::CommandLine,
                         true,
@@ -374,7 +381,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 if p.is_last_set() && !trailing_values {
                     return Err(ClapError::unknown_argument(
                         self.cmd,
-                        arg_os.to_str_lossy().into_owned(),
+                        arg_os.display().to_string(),
                         None,
                         Usage::new(self.cmd).create_usage_with_title(&[]),
                     ));
@@ -395,7 +402,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 let append = self.has_val_groups(matcher, p);
                 self.add_val_to_arg(
                     p,
-                    &arg_os,
+                    arg_os.to_value_os(),
                     matcher,
                     ValueSource::CommandLine,
                     append,
@@ -412,9 +419,9 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 valid_arg_found = true;
             } else if self.cmd.is_allow_external_subcommands_set() {
                 // Get external subcommand name
-                let sc_name = match arg_os.to_str() {
-                    Some(s) => s.to_string(),
-                    None => {
+                let sc_name = match arg_os.to_value() {
+                    Ok(s) => s.to_string(),
+                    Err(_) => {
                         return Err(ClapError::invalid_utf8(
                             self.cmd,
                             Usage::new(self.cmd).create_usage_with_title(&[]),
@@ -425,7 +432,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 // Collect the external subcommand args
                 let mut sc_m = ArgMatcher::new(self.cmd);
 
-                while let Some((v, _)) = it.next() {
+                for v in raw_args.remaining(&mut args_cursor) {
                     let allow_invalid_utf8 = self
                         .cmd
                         .is_allow_invalid_utf8_for_external_subcommands_set();
@@ -466,7 +473,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 .expect(INTERNAL_ERROR_MSG)
                 .get_name()
                 .to_owned();
-            self.parse_subcommand(&sc_name, matcher, it, keep_state)?;
+            self.parse_subcommand(&sc_name, matcher, raw_args, args_cursor, keep_state)?;
         }
 
         Validator::new(self).validate(parse_state, matcher, trailing_values)
@@ -474,23 +481,28 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
 
     fn match_arg_error(
         &self,
-        arg_os: &RawOsStr,
+        arg_os: &clap_lex::ParsedArg<'_>,
         valid_arg_found: bool,
         trailing_values: bool,
     ) -> ClapError {
         // If argument follows a `--`
         if trailing_values {
             // If the arg matches a subcommand name, or any of its aliases (if defined)
-            if self.possible_subcommand(arg_os, valid_arg_found).is_some() {
+            if self
+                .possible_subcommand(arg_os.to_value(), valid_arg_found)
+                .is_some()
+            {
                 return ClapError::unnecessary_double_dash(
                     self.cmd,
-                    arg_os.to_str_lossy().into_owned(),
+                    arg_os.display().to_string(),
                     Usage::new(self.cmd).create_usage_with_title(&[]),
                 );
             }
         }
-        let candidates =
-            suggestions::did_you_mean(&arg_os.to_str_lossy(), self.cmd.all_subcommand_names());
+        let candidates = suggestions::did_you_mean(
+            &arg_os.display().to_string(),
+            self.cmd.all_subcommand_names(),
+        );
         // If the argument looks like a subcommand.
         if !candidates.is_empty() {
             let candidates: Vec<_> = candidates
@@ -499,7 +511,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 .collect();
             return ClapError::invalid_subcommand(
                 self.cmd,
-                arg_os.to_str_lossy().into_owned(),
+                arg_os.display().to_string(),
                 candidates.join(" or "),
                 self.cmd
                     .get_bin_name()
@@ -513,7 +525,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         {
             return ClapError::unrecognized_subcommand(
                 self.cmd,
-                arg_os.to_str_lossy().into_owned(),
+                arg_os.display().to_string(),
                 self.cmd
                     .get_bin_name()
                     .unwrap_or_else(|| self.cmd.get_name())
@@ -522,15 +534,20 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         }
         ClapError::unknown_argument(
             self.cmd,
-            arg_os.to_str_lossy().into_owned(),
+            arg_os.display().to_string(),
             None,
             Usage::new(self.cmd).create_usage_with_title(&[]),
         )
     }
 
     // Checks if the arg matches a subcommand name, or any of its aliases (if defined)
-    fn possible_subcommand(&self, arg_os: &RawOsStr, valid_arg_found: bool) -> Option<&str> {
-        debug!("Parser::possible_subcommand: arg={:?}", arg_os);
+    fn possible_subcommand(
+        &self,
+        arg: Result<&str, &RawOsStr>,
+        valid_arg_found: bool,
+    ) -> Option<&str> {
+        debug!("Parser::possible_subcommand: arg={:?}", arg);
+        let arg = arg.ok()?;
 
         if !(self.cmd.is_args_conflicts_with_subcommands_set() && valid_arg_found) {
             if self.cmd.is_infer_subcommands_set() {
@@ -539,7 +556,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 let v = self
                     .cmd
                     .all_subcommand_names()
-                    .filter(|s| RawOsStr::from_str(s).starts_with_os(arg_os))
+                    .filter(|s| s.starts_with(arg))
                     .collect::<Vec<_>>();
 
                 if v.len() == 1 {
@@ -549,7 +566,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 // If there is any ambiguity, fallback to non-infer subcommand
                 // search.
             }
-            if let Some(sc) = self.cmd.find_subcommand(arg_os) {
+            if let Some(sc) = self.cmd.find_subcommand(arg) {
                 return Some(sc.get_name());
             }
         }
@@ -557,21 +574,18 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
     }
 
     // Checks if the arg matches a long flag subcommand name, or any of its aliases (if defined)
-    fn possible_long_flag_subcommand(&self, arg_os: &RawOsStr) -> Option<&str> {
-        debug!("Parser::possible_long_flag_subcommand: arg={:?}", arg_os);
+    fn possible_long_flag_subcommand(&self, arg: &str) -> Option<&str> {
+        debug!("Parser::possible_long_flag_subcommand: arg={:?}", arg);
         if self.cmd.is_infer_subcommands_set() {
             let options = self
                 .cmd
                 .get_subcommands()
                 .fold(Vec::new(), |mut options, sc| {
                     if let Some(long) = sc.get_long_flag() {
-                        if RawOsStr::from_str(long).starts_with_os(arg_os) {
+                        if long.starts_with(arg) {
                             options.push(long);
                         }
-                        options.extend(
-                            sc.get_all_aliases()
-                                .filter(|alias| RawOsStr::from_str(alias).starts_with_os(arg_os)),
-                        )
+                        options.extend(sc.get_all_aliases().filter(|alias| alias.starts_with(arg)))
                     }
                     options
                 });
@@ -580,17 +594,20 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
             }
 
             for sc in options {
-                if sc == arg_os {
+                if sc == arg {
                     return Some(sc);
                 }
             }
-        } else if let Some(sc_name) = self.cmd.find_long_subcmd(arg_os) {
+        } else if let Some(sc_name) = self.cmd.find_long_subcmd(arg) {
             return Some(sc_name);
         }
         None
     }
 
-    fn parse_help_subcommand(&self, cmds: &[OsString]) -> ClapResult<ParseResult> {
+    fn parse_help_subcommand(
+        &self,
+        cmds: impl Iterator<Item = &'cmd OsStr>,
+    ) -> ClapResult<ParseResult> {
         debug!("Parser::parse_help_subcommand");
 
         let mut bin_name = self
@@ -602,7 +619,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         let mut sc = {
             let mut sc = self.cmd.clone();
 
-            for cmd in cmds.iter() {
+            for cmd in cmds {
                 sc = if let Some(c) = sc.find_subcommand(cmd) {
                     c
                 } else if let Some(c) = sc.find_subcommand(&cmd.to_string_lossy()) {
@@ -633,32 +650,42 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         Err(parser.help_err(true))
     }
 
-    fn is_new_arg(&self, next: &RawOsStr, current_positional: &Arg) -> bool {
+    fn is_new_arg(&self, next: &clap_lex::ParsedArg<'_>, current_positional: &Arg) -> bool {
+        #![allow(clippy::needless_bool)] // Prefer consistent if/else-if ladder
+
         debug!(
             "Parser::is_new_arg: {:?}:{:?}",
-            next, current_positional.name
+            next.to_value_os(),
+            current_positional.name
         );
 
         if self.cmd.is_allow_hyphen_values_set()
             || self.cmd[&current_positional.id].is_allow_hyphen_values_set()
-            || (self.cmd.is_allow_negative_numbers_set()
-                && next.to_str_lossy().parse::<f64>().is_ok())
+            || (self.cmd.is_allow_negative_numbers_set() && next.is_number())
         {
             // If allow hyphen, this isn't a new arg.
             debug!("Parser::is_new_arg: Allow hyphen");
             false
-        } else if next.starts_with("--") {
-            // If this is a long flag, this is a new arg.
+        } else if next.is_escape() {
+            // Ensure we don't assuming escapes are long args
             debug!("Parser::is_new_arg: -- found");
-            true
-        } else if next.starts_with("-") {
+            false
+        } else if next.is_stdio() {
+            // Ensure we don't assume stdio is a short arg
             debug!("Parser::is_new_arg: - found");
+            false
+        } else if next.is_long() {
+            // If this is a long flag, this is a new arg.
+            debug!("Parser::is_new_arg: --<something> found");
+            true
+        } else if next.is_short() {
             // If this is a short flag, this is a new arg. But a singe '-' by
             // itself is a value and typically means "stdin" on unix systems.
-            next.raw_len() != 1
+            debug!("Parser::is_new_arg: -<something> found");
+            true
         } else {
-            debug!("Parser::is_new_arg: value");
             // Nothing special, this is a value.
+            debug!("Parser::is_new_arg: value");
             false
         }
     }
@@ -667,7 +694,8 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         &mut self,
         sc_name: &str,
         matcher: &mut ArgMatcher,
-        it: &mut Input,
+        raw_args: &mut clap_lex::RawArgs,
+        args_cursor: clap_lex::ArgCursor,
         keep_state: bool,
     ) -> ClapResult<()> {
         debug!("Parser::parse_subcommand");
@@ -691,7 +719,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                     p.flag_subcmd_at = self.flag_subcmd_at;
                     p.flag_subcmd_skip = self.flag_subcmd_skip;
                 }
-                if let Err(error) = p.get_matches_with(&mut sc_matcher, it) {
+                if let Err(error) = p.get_matches_with(&mut sc_matcher, raw_args, args_cursor) {
                     if partial_parsing_enabled {
                         debug!(
                             "Parser::parse_subcommand: ignored error in subcommand {}: {:?}",
@@ -804,7 +832,8 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
     fn parse_long_arg(
         &mut self,
         matcher: &mut ArgMatcher,
-        long_arg: &RawOsStr,
+        long_arg: Result<&str, &RawOsStr>,
+        long_value: Option<&RawOsStr>,
         parse_state: &ParseState,
         valid_arg_found: &mut bool,
         trailing_values: bool,
@@ -823,31 +852,31 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         debug!("Parser::parse_long_arg: cur_idx:={}", self.cur_idx.get());
 
         debug!("Parser::parse_long_arg: Does it contain '='...");
+        let long_arg = match long_arg {
+            Ok(long_arg) => long_arg,
+            Err(long_arg) => {
+                return ParseResult::NoMatchingArg {
+                    arg: long_arg.to_str_lossy().into_owned(),
+                };
+            }
+        };
         if long_arg.is_empty() {
+            debug_assert!(long_value.is_none(), "{:?}", long_value);
             return ParseResult::NoArg;
         }
-        let (arg, val) = if let Some(index) = long_arg.find("=") {
-            let (p0, p1) = long_arg.split_at(index);
-            debug!("Yes '{:?}'", p1);
-            (p0, Some(p1))
-        } else {
-            debug!("No");
-            (long_arg, None)
-        };
 
-        let opt = if let Some(opt) = self.cmd.get_keymap().get(&*arg.to_os_str()) {
+        let opt = if let Some(opt) = self.cmd.get_keymap().get(long_arg) {
             debug!(
                 "Parser::parse_long_arg: Found valid opt or flag '{}'",
                 opt.to_string()
             );
             Some(opt)
         } else if self.cmd.is_infer_long_args_set() {
-            let arg_str = arg.to_str_lossy();
             self.cmd.get_arguments().find(|a| {
-                a.long.map_or(false, |long| long.starts_with(&*arg_str))
+                a.long.map_or(false, |long| long.starts_with(long_arg))
                     || a.aliases
                         .iter()
-                        .any(|(alias, _)| alias.starts_with(&*arg_str))
+                        .any(|(alias, _)| alias.starts_with(long_arg))
             })
         } else {
             None
@@ -859,10 +888,11 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
             if opt.is_takes_value_set() {
                 debug!(
                     "Parser::parse_long_arg: Found an opt with value '{:?}'",
-                    &val
+                    &long_value
                 );
-                self.parse_opt(val, opt, matcher, trailing_values)
-            } else if let Some(rest) = val {
+                let has_eq = long_value.is_some();
+                self.parse_opt(long_value, opt, matcher, trailing_values, has_eq)
+            } else if let Some(rest) = long_value {
                 let required = self.cmd.required_graph();
                 debug!("Parser::parse_long_arg: Got invalid literal `{:?}`", rest);
                 let used: Vec<Id> = matcher
@@ -880,19 +910,21 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                     used,
                     arg: opt.to_string(),
                 }
-            } else if let Some(parse_result) = self.check_for_help_and_version_str(arg) {
+            } else if let Some(parse_result) =
+                self.check_for_help_and_version_str(RawOsStr::from_str(long_arg))
+            {
                 parse_result
             } else {
                 debug!("Parser::parse_long_arg: Presence validated");
                 self.parse_flag(opt, matcher)
             }
-        } else if let Some(sc_name) = self.possible_long_flag_subcommand(arg) {
+        } else if let Some(sc_name) = self.possible_long_flag_subcommand(long_arg) {
             ParseResult::FlagSubCommand(sc_name.to_string())
         } else if self.cmd.is_allow_hyphen_values_set() {
             ParseResult::MaybeHyphenValue
         } else {
             ParseResult::NoMatchingArg {
-                arg: arg.to_str_lossy().into_owned(),
+                arg: long_arg.to_owned(),
             }
         }
     }
@@ -900,7 +932,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
     fn parse_short_arg(
         &mut self,
         matcher: &mut ArgMatcher,
-        short_arg: &RawOsStr,
+        mut short_arg: clap_lex::ShortFlags<'_>,
         parse_state: &ParseState,
         // change this to possible pos_arg when removing the usage of &mut Parser.
         pos_counter: usize,
@@ -908,14 +940,15 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         trailing_values: bool,
     ) -> ParseResult {
         debug!("Parser::parse_short_arg: short_arg={:?}", short_arg);
-        let arg = short_arg.to_str_lossy();
 
         #[allow(clippy::blocks_in_if_conditions)]
-        if self.cmd.is_allow_negative_numbers_set() && arg.parse::<f64>().is_ok() {
+        if self.cmd.is_allow_negative_numbers_set() && short_arg.is_number() {
             debug!("Parser::parse_short_arg: negative number");
             return ParseResult::MaybeHyphenValue;
         } else if self.cmd.is_allow_hyphen_values_set()
-            && arg.chars().any(|c| !self.cmd.contains_short(c))
+            && short_arg
+                .clone()
+                .any(|c| !c.map(|c| self.cmd.contains_short(c)).unwrap_or_default())
         {
             debug!("Parser::parse_short_args: contains non-short flag");
             return ParseResult::MaybeHyphenValue;
@@ -943,7 +976,22 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
 
         let skip = self.flag_subcmd_skip;
         self.flag_subcmd_skip = 0;
-        for c in arg.chars().skip(skip) {
+        let res = short_arg.advance_by(skip);
+        debug_assert_eq!(
+            res,
+            Ok(()),
+            "tracking of `flag_subcmd_skip` is off for `{:?}`",
+            short_arg
+        );
+        while let Some(c) = short_arg.next_flag() {
+            let c = match c {
+                Ok(c) => c,
+                Err(rest) => {
+                    return ParseResult::NoMatchingArg {
+                        arg: format!("-{}", rest.to_str_lossy()),
+                    };
+                }
+            };
             debug!("Parser::parse_short_arg:iter:{}", c);
 
             // update each index because `-abcd` is four indices to clap
@@ -974,7 +1022,9 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 }
 
                 // Check for trailing concatenated value
-                let val = short_arg.split_once(c).expect(INTERNAL_ERROR_MSG).1;
+                //
+                // Cloning the iterator, so we rollback if it isn't there.
+                let val = short_arg.clone().next_value_os().unwrap_or_default();
                 debug!(
                     "Parser::parse_short_arg:iter:{}: val={:?} (bytes), val={:?} (ascii), short_arg={:?}",
                     c, val, val.as_raw_bytes(), short_arg
@@ -988,7 +1038,12 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                 //
                 // e.g. `-xvf`, when require_equals && x.min_vals == 0, we don't
                 // consume the `vf`, even if it's provided as value.
-                match self.parse_opt(val, opt, matcher, trailing_values) {
+                let (val, has_eq) = if let Some(val) = val.and_then(|v| v.strip_prefix('=')) {
+                    (Some(val), true)
+                } else {
+                    (val, false)
+                };
+                match self.parse_opt(val, opt, matcher, trailing_values, has_eq) {
                     ParseResult::AttachedValueNotConsumed => continue,
                     x => return x,
                 }
@@ -997,17 +1052,12 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
             return if let Some(sc_name) = self.cmd.find_short_subcmd(c) {
                 debug!("Parser::parse_short_arg:iter:{}: subcommand={}", c, sc_name);
                 let name = sc_name.to_string();
-                let done_short_args = {
-                    let cur_idx = self.cur_idx.get();
-                    // Get the index of the previously saved flag subcommand in the group of flags (if exists).
-                    // If it is a new flag subcommand, then the formentioned index should be the current one
-                    // (ie. `cur_idx`), and should be registered.
-                    let at = *self.flag_subcmd_at.get_or_insert(cur_idx);
-                    // If we are done, then the difference of indices (cur_idx - at) should be (end - at) which
-                    // should equal to (arg.len() - 1),
-                    // where `end` is the index of the end of the group.
-                    cur_idx - at == arg.len() - 1
-                };
+                // Get the index of the previously saved flag subcommand in the group of flags (if exists).
+                // If it is a new flag subcommand, then the formentioned index should be the current one
+                // (ie. `cur_idx`), and should be registered.
+                let cur_idx = self.cur_idx.get();
+                self.flag_subcmd_at.get_or_insert(cur_idx);
+                let done_short_args = short_arg.is_empty();
                 if done_short_args {
                     self.flag_subcmd_at = None;
                 }
@@ -1027,14 +1077,13 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
         opt: &Arg<'help>,
         matcher: &mut ArgMatcher,
         trailing_values: bool,
+        has_eq: bool,
     ) -> ParseResult {
         debug!(
-            "Parser::parse_opt; opt={}, val={:?}",
-            opt.name, attached_value
+            "Parser::parse_opt; opt={}, val={:?}, has_eq={:?}",
+            opt.name, attached_value, has_eq
         );
         debug!("Parser::parse_opt; opt.settings={:?}", opt.settings);
-        // has_eq: --flag=value
-        let has_eq = matches!(attached_value, Some(fv) if fv.starts_with("="));
 
         debug!("Parser::parse_opt; Checking for val...");
         // require_equals is set, but no '=' is provided, try throwing error.
@@ -1064,14 +1113,7 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
                     arg: opt.to_string(),
                 }
             }
-        } else if let Some(fv) = attached_value {
-            let v = fv.strip_prefix("=").unwrap_or(fv);
-            debug!("Found - {:?}, len: {}", v, v.raw_len());
-            debug!(
-                "Parser::parse_opt: {:?} contains '='...{:?}",
-                fv,
-                fv.starts_with("=")
-            );
+        } else if let Some(v) = attached_value {
             self.inc_occurrence_of_arg(matcher, opt);
             self.add_val_to_arg(
                 opt,
@@ -1531,49 +1573,6 @@ impl<'help, 'cmd> Parser<'help, 'cmd> {
     }
 }
 
-#[derive(Debug)]
-pub(crate) struct Input {
-    items: Vec<OsString>,
-    cursor: usize,
-}
-
-impl<I, T> From<I> for Input
-where
-    I: Iterator<Item = T>,
-    T: Into<OsString> + Clone,
-{
-    fn from(val: I) -> Self {
-        Self {
-            items: val.map(|x| x.into()).collect(),
-            cursor: 0,
-        }
-    }
-}
-
-impl Input {
-    pub(crate) fn next(&mut self) -> Option<(&OsStr, &[OsString])> {
-        if self.cursor >= self.items.len() {
-            None
-        } else {
-            let current = &self.items[self.cursor];
-            self.cursor += 1;
-            let remaining = &self.items[self.cursor..];
-            Some((current, remaining))
-        }
-    }
-
-    /// Insert some items to the Input items just after current parsing cursor.
-    /// Usually used by replaced items recovering.
-    pub(crate) fn insert(&mut self, insert_items: &[&str]) {
-        self.items = insert_items
-            .iter()
-            .map(OsString::from)
-            .chain(self.items.drain(self.cursor..))
-            .collect();
-        self.cursor = 0;
-    }
-}
-
 #[derive(Debug)]
 pub(crate) enum ParseState {
     ValuesDone,


Test patch:
diff --git a/clap_lex/tests/lexer.rs b/clap_lex/tests/lexer.rs
new file mode 100644
index 00000000000..bdfae2fd91b
--- /dev/null
+++ b/clap_lex/tests/lexer.rs
@@ -0,0 +1,21 @@
+#[test]
+fn insert() {
+    let mut raw = clap_lex::RawArgs::new(["bin", "a", "b", "c"]);
+    let mut cursor = raw.cursor();
+
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("a")));
+    raw.insert(&cursor, &["1", "2", "3"]);
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("1")));
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("2")));
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("3")));
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("b")));
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("c")));
+
+    let mut cursor = raw.cursor();
+    let rest = raw
+        .remaining(&mut cursor)
+        .map(|s| s.to_string_lossy())
+        .collect::<Vec<_>>();
+    assert_eq!(rest, vec!["bin", "a", "1", "2", "3", "b", "c"]);
+}
diff --git a/clap_lex/tests/parsed.rs b/clap_lex/tests/parsed.rs
new file mode 100644
index 00000000000..9789011c073
--- /dev/null
+++ b/clap_lex/tests/parsed.rs
@@ -0,0 +1,190 @@
+// Despite our design philosophy being to support completion generation, we aren't considering `-`
+// the start of a long because there is no valid value to return.
+#[test]
+fn to_long_stdio() {
+    let raw = clap_lex::RawArgs::new(["bin", "-"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(!next.is_long());
+
+    assert_eq!(next.to_long(), None);
+}
+
+#[test]
+fn to_long_escape() {
+    let raw = clap_lex::RawArgs::new(["bin", "--"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_long());
+
+    let (key, value) = next.to_long().unwrap();
+    assert_eq!(key, Ok(""));
+    assert_eq!(value, None);
+}
+
+#[test]
+fn to_long_no_value() {
+    let raw = clap_lex::RawArgs::new(["bin", "--long"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_long());
+
+    let (key, value) = next.to_long().unwrap();
+    assert_eq!(key, Ok("long"));
+    assert_eq!(value, None);
+}
+
+#[test]
+fn to_long_with_empty_value() {
+    let raw = clap_lex::RawArgs::new(["bin", "--long="]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_long());
+
+    let (key, value) = next.to_long().unwrap();
+    assert_eq!(key, Ok("long"));
+    assert_eq!(value, Some(clap_lex::RawOsStr::from_str("")));
+}
+
+#[test]
+fn to_long_with_value() {
+    let raw = clap_lex::RawArgs::new(["bin", "--long=hello"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_long());
+
+    let (key, value) = next.to_long().unwrap();
+    assert_eq!(key, Ok("long"));
+    assert_eq!(value, Some(clap_lex::RawOsStr::from_str("hello")));
+}
+
+#[test]
+fn to_short_stdio() {
+    let raw = clap_lex::RawArgs::new(["bin", "-"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_short());
+
+    let mut shorts = next.to_short().unwrap();
+    assert_eq!(shorts.next_value_os(), None);
+}
+
+#[test]
+fn to_short_escape() {
+    let raw = clap_lex::RawArgs::new(["bin", "--"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(!next.is_short());
+
+    assert!(next.to_short().is_none());
+}
+
+#[test]
+fn to_short_long() {
+    let raw = clap_lex::RawArgs::new(["bin", "--long"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(!next.is_short());
+
+    assert!(next.to_short().is_none());
+}
+
+#[test]
+fn to_short() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_short());
+
+    let shorts = next.to_short().unwrap();
+    let actual: String = shorts.map(|s| s.unwrap()).collect();
+    assert_eq!(actual, "short");
+}
+
+#[test]
+fn is_negative_number() {
+    let raw = clap_lex::RawArgs::new(["bin", "-10.0"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_number());
+}
+
+#[test]
+fn is_positive_number() {
+    let raw = clap_lex::RawArgs::new(["bin", "10.0"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_number());
+}
+
+#[test]
+fn is_not_number() {
+    let raw = clap_lex::RawArgs::new(["bin", "--10.0"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(!next.is_number());
+}
+
+#[test]
+fn is_stdio() {
+    let raw = clap_lex::RawArgs::new(["bin", "-"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_stdio());
+}
+
+#[test]
+fn is_not_stdio() {
+    let raw = clap_lex::RawArgs::new(["bin", "--"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(!next.is_stdio());
+}
+
+#[test]
+fn is_escape() {
+    let raw = clap_lex::RawArgs::new(["bin", "--"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(next.is_escape());
+}
+
+#[test]
+fn is_not_escape() {
+    let raw = clap_lex::RawArgs::new(["bin", "-"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+
+    assert!(!next.is_escape());
+}
diff --git a/clap_lex/tests/shorts.rs b/clap_lex/tests/shorts.rs
new file mode 100644
index 00000000000..a583a883fa2
--- /dev/null
+++ b/clap_lex/tests/shorts.rs
@@ -0,0 +1,198 @@
+#[test]
+fn iter() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let shorts = next.to_short().unwrap();
+
+    let actual: String = shorts.map(|s| s.unwrap()).collect();
+    assert_eq!(actual, "short");
+}
+
+#[test]
+fn next_flag() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    let mut actual = String::new();
+    actual.push(shorts.next_flag().unwrap().unwrap());
+    actual.push(shorts.next_flag().unwrap().unwrap());
+    actual.push(shorts.next_flag().unwrap().unwrap());
+    actual.push(shorts.next_flag().unwrap().unwrap());
+    actual.push(shorts.next_flag().unwrap().unwrap());
+    assert_eq!(shorts.next_flag(), None);
+
+    assert_eq!(actual, "short");
+}
+
+#[test]
+fn next_value_os() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    let actual = shorts.next_value_os().unwrap().to_str_lossy();
+
+    assert_eq!(actual, "short");
+}
+
+#[test]
+fn next_flag_with_value() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    assert_eq!(shorts.next_flag().unwrap().unwrap(), 's');
+    let actual = shorts.next_value_os().unwrap().to_str_lossy();
+
+    assert_eq!(actual, "hort");
+}
+
+#[test]
+fn next_flag_with_no_value() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    assert_eq!(shorts.next_flag().unwrap().unwrap(), 's');
+    assert_eq!(shorts.next_flag().unwrap().unwrap(), 'h');
+    assert_eq!(shorts.next_flag().unwrap().unwrap(), 'o');
+    assert_eq!(shorts.next_flag().unwrap().unwrap(), 'r');
+    assert_eq!(shorts.next_flag().unwrap().unwrap(), 't');
+
+    assert_eq!(shorts.next_value_os(), None);
+}
+
+#[test]
+fn advance_by_nothing() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    assert_eq!(shorts.advance_by(0), Ok(()));
+
+    let actual: String = shorts.map(|s| s.unwrap()).collect();
+    assert_eq!(actual, "short");
+}
+
+#[test]
+fn advance_by_nothing_with_nothing() {
+    let raw = clap_lex::RawArgs::new(["bin", "-"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    assert_eq!(shorts.advance_by(0), Ok(()));
+
+    let actual: String = shorts.map(|s| s.unwrap()).collect();
+    assert_eq!(actual, "");
+}
+
+#[test]
+fn advance_by_something() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    assert_eq!(shorts.advance_by(2), Ok(()));
+
+    let actual: String = shorts.map(|s| s.unwrap()).collect();
+    assert_eq!(actual, "ort");
+}
+
+#[test]
+fn advance_by_out_of_bounds() {
+    let raw = clap_lex::RawArgs::new(["bin", "-short"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+
+    assert_eq!(shorts.advance_by(2000), Err(5));
+
+    let actual: String = shorts.map(|s| s.unwrap()).collect();
+    assert_eq!(actual, "");
+}
+
+#[test]
+fn is_empty() {
+    let raw = clap_lex::RawArgs::new(["bin", "-"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let shorts = next.to_short().unwrap();
+
+    assert!(shorts.is_empty());
+}
+
+#[test]
+fn is_not_empty() {
+    let raw = clap_lex::RawArgs::new(["bin", "-hello"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let shorts = next.to_short().unwrap();
+
+    assert!(!shorts.is_empty());
+}
+
+#[test]
+fn is_partial_not_empty() {
+    let raw = clap_lex::RawArgs::new(["bin", "-hello"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+    shorts.advance_by(1).unwrap();
+
+    assert!(!shorts.is_empty());
+}
+
+#[test]
+fn is_exhausted_empty() {
+    let raw = clap_lex::RawArgs::new(["bin", "-"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let mut shorts = next.to_short().unwrap();
+    shorts.advance_by(20000).unwrap_err();
+
+    assert!(shorts.is_empty());
+}
+
+#[test]
+fn is_number() {
+    let raw = clap_lex::RawArgs::new(["bin", "-1.0"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let shorts = next.to_short().unwrap();
+
+    assert!(shorts.is_number());
+}
+
+#[test]
+fn is_not_number() {
+    let raw = clap_lex::RawArgs::new(["bin", "-hello"]);
+    let mut cursor = raw.cursor();
+    assert_eq!(raw.next_os(&mut cursor), Some(std::ffi::OsStr::new("bin")));
+    let next = raw.next(&mut cursor).unwrap();
+    let shorts = next.to_short().unwrap();
+
+    assert!(!shorts.is_number());
+}