Repository: winnow-rs/winnow (Rust)

Issue:
ParseError offset out of bounds of input stream
### Please complete the following tasks

- [X] I have searched the [discussions](https://github.com/winnow-rs/winnow/discussions)
- [X] I have searched the [open](https://github.com/winnow-rs/winnow/issues) and [rejected](https://github.com/winnow-rs/winnow/issues?q=is%3Aissue+label%3AS-wont-fix+is%3Aclosed) issues

### rust version

rustc 1.72.1 (d5c2e9c34 2023-09-13)

### winnow version

0.5.15

### Minimal reproducible code
Also available as a [repo](https://github.com/adamchalmers/winnow-offset-bug-repro)
```rust
use winnow::{prelude::*, token::any};

fn main() {
    let tokens = [
        Token {
            token_type: TokenType::Whitespace,
            value: " ".to_owned(),
        },
        // If this token was changed to TokenType::Word and value "asdf",
        // the parser would succeed.
        Token {
            token_type: TokenType::Keyword,
            value: "let".to_owned(),
        },
    ];
    let result = declaration.parse(&mut tokens.as_slice());
    eprintln!("{result:#?}");
    let err = result.unwrap_err();

    // Expectation: I can find the bad token (the token which caused the error
    // by indexing err.offset() into err.input().
    // However, this actually gives an index-out-of-bounds panic, because
    // the offset is 32 and the input slice only has length 2.
    let stream = err.input();
    let bad_token = &stream[err.offset()];
    eprintln!("Bad token: {bad_token:?}");
}

/* The stream is a vec of custom tokens. */
type TokenSlice<'slice, 'input> = &'slice mut &'input [Token];

#[derive(Debug, PartialEq, Eq, Copy, Clone)]
pub enum TokenType {
    Word,
    Keyword,
    Whitespace,
}

#[derive(Debug, PartialEq, Eq, Clone)]
pub struct Token {
    pub token_type: TokenType,
    pub value: String,
}

/* Parsers */

/// Parse some whitespace (i.e. at least one whitespace token)
fn whitespace(i: TokenSlice) -> PResult<Token> {
    any.verify(|token: &Token| token.token_type == TokenType::Whitespace)
        .parse_next(i)
}

/// Parse a variable/constant declaration.
pub fn declaration(i: TokenSlice) -> PResult<()> {
    whitespace.parse_next(i)?;
    identifier.parse_next(i)?;
    Ok(())
}

/// Parse an identifier (name of a constant/variable/function)
fn identifier(i: TokenSlice) -> PResult<Token> {
    any.verify(|token: &Token| token.token_type == TokenType::Word)
        .parse_next(i)
}

```


### Steps to reproduce the bug with the above code

`cargo run`

### Actual Behaviour

When I run `cargo run`:

- the `declaration` parser fails (expected)
- the failing parse returns a `ParseError` (expected)
- the `ParseError` returns the stream of tokens in its `input` field, with all 2 of the input tokens (expected)
- the `ParseError` returns a `.offset()` of 32, which is massively out of range from the input stream of length 2 (unexpected)
- this causes an "index out of bounds" error when trying to find the token which caused the error (unexpected)

### Expected Behaviour

The ParseError::offset() method should return a number between 0 and the length of the input stream.

### Additional Context

I am using an input stream which is a vec of custom token enums. i.e. `TokenSlice<'slice, 'input> = &'slice mut &'input [Token];`


Discussion:
Thanks Ed!



Patch:
diff --git a/src/stream/mod.rs b/src/stream/mod.rs
index 864db5c99..9d98c761d 100644
--- a/src/stream/mod.rs
+++ b/src/stream/mod.rs
@@ -1300,7 +1300,7 @@ impl<'a, T> Offset for &'a [T] {
             fst <= snd,
             "`Offset::offset_to` only accepts slices of `self`"
         );
-        snd as usize - fst as usize
+        (snd as usize - fst as usize) / crate::lib::std::mem::size_of::<T>()
     }
 }
 


Test patch:
diff --git a/src/stream/tests.rs b/src/stream/tests.rs
index e653ad9e0..2dbb807b9 100644
--- a/src/stream/tests.rs
+++ b/src/stream/tests.rs
@@ -114,3 +114,18 @@ fn test_partial_complete() {
     i.restore_partial(incomplete_state);
     assert!(i.is_partial(), "incomplete stream state should be restored");
 }
+
+#[test]
+fn test_custom_slice() {
+    type Token = usize;
+    type TokenSlice<'i> = &'i [Token];
+
+    let mut tokens: TokenSlice<'_> = &[1, 2, 3, 4];
+
+    let input = &mut tokens;
+    let start = input.checkpoint();
+    let _ = input.next_token();
+    let _ = input.next_token();
+    let offset = input.offset_from(&start);
+    assert_eq!(offset, 2);
+}