Repository: nushell/nushell (language: rust)

Issue:
Completion fails after `go work use`
### Describe the bug

Type `go work use <tab>`.  nushell will report NO RECORDS FOUND.  I would expect nushell to fallback on filepath completion.  If I type `xxx yyy zzz <tab>`, I get the expected filepath completion.

I have external completers commented out entirely.  So this is just using native completion.  But even when I have external completers enabled (using the cookbook recipe that uses fish, zoxide, and carapace) I get the same behavior.  When the external completer is enabled, this should be resolving to a call to carapace, but I've tried directly calling `carapace go nushell go work use ""` and I get back carapace's file completion, as expected, so I suspect carapace isn't even being invoked.

Completing `go <tab>` and `go work <tab>` work as expected.

Interestingly, if I type `go work use ~/<tab>` I sort of do get file completions, but the list presented is colored incorrectly, and I can't fuzzy match against the options presented.  

So, when I type `bat ~/<tab>`, I see this:

![Image](https://github.com/user-attachments/assets/0423de6f-b841-48a6-9d44-2c900a047635)

And if I then type `.`, it narrows the list of completions, like this:

![Image](https://github.com/user-attachments/assets/6d109c50-caf7-444c-8121-20314a0457ce)

But, if I type `go work use ~/<tab>`, I get this:

![Image](https://github.com/user-attachments/assets/165e712a-db29-48ce-a85a-1d97213793ae)

And if I then type `.` to narrow that list, I get:

![Image](https://github.com/user-attachments/assets/c61d6716-b7c2-48d8-932e-05e0b32339b4)

Also, doing `go work use ~<tab>` also just results in "NO RECORDS FOUND"

### How to reproduce

1. Type `go work use <tab>`

### Expected behavior

I expected nu to present file path completion.  When I have my external completer configured, I expect it to present carapace's file completion, and when the external completer is removed, I expect nushell's file completion.

### Configuration

| key                | value                                          |
| ------------------ | ---------------------------------------------- |
| version            | 0.101.0                                        |
| major              | 0                                              |
| minor              | 101                                            |
| patch              | 0                                              |
| branch             |                                                |
| commit_hash        |                                                |
| build_os           | macos-aarch64                                  |
| build_target       | aarch64-apple-darwin                           |
| rust_version       | rustc 1.83.0 (90b35a623 2024-11-26) (Homebrew) |
| cargo_version      | cargo 1.83.0                                   |
| build_time         | 2024-12-22 14:10:19 +00:00                     |
| build_rust_channel | release                                        |
| allocator          | mimalloc                                       |
| features           | default, sqlite, trash                         |
| installed_plugins  |                                                |


Discussion / hints:
There's a small chance this is happening because we special-case completions for `use`.

This is what I think is happening, although I haven't tested yet. [This line](https://github.com/nushell/nushell/blob/3f5ebd75b6e1506f06509bc29a4807be1bc25049/crates/nu-cli/src/completions/completer.rs#L263) checks `if prev_expr_str == b"use"`. If `go work use` were either a custom command or a declared extern, then the previous expression would've been the entire string `go work use`. However, since `go` is an external and Nushell doesn't know about its subcommands, `work` and `use` will be separate bareword strings, and so `prev_expr_str` will just be `use` here. This triggers our special completions for `use`, which should only show you directories and .nu files.

Edit: yup, I can reproduce this even with `echo use ./<TAB>`:

![Image](https://github.com/user-attachments/assets/7cb709fd-cb6d-4052-ba45-512d6fe019c1)
I think this could be fixed in one of the following ways:
1. Add an extra check to make sure we're currently looking at a resolved command call. That way, we'll know that `use`/`source-env`/`ls` is the entire command, not just the last string.
2. Instead of a `prev_expr_str`, make a string holding the entire pipeline element up to that point. We'd have to strip out whitespace though. Perhaps that's fine, because we already seem to require people to type `overlay use` with a single space in between, rather than `overlay    use` or something, to get completions.

Fix patch:
diff --git a/crates/nu-cli/src/completions/attribute_completions.rs b/crates/nu-cli/src/completions/attribute_completions.rs
index 237b5bcbb3b2b..4d2982149b3b6 100644
--- a/crates/nu-cli/src/completions/attribute_completions.rs
+++ b/crates/nu-cli/src/completions/attribute_completions.rs
@@ -17,23 +17,15 @@ impl Completer for AttributeCompletion {
         &mut self,
         working_set: &StateWorkingSet,
         _stack: &Stack,
-        _prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        let partial = working_set.get_span_contents(span);
-        let mut matcher = NuMatcher::new(String::from_utf8_lossy(partial), options.clone());
+        let mut matcher = NuMatcher::new(prefix, options);
 
-        let attr_commands = working_set.find_commands_by_predicate(
-            |s| {
-                s.strip_prefix(b"attr ")
-                    .map(String::from_utf8_lossy)
-                    .is_some_and(|name| matcher.matches(&name))
-            },
-            true,
-        );
+        let attr_commands =
+            working_set.find_commands_by_predicate(|s| s.starts_with(b"attr "), true);
 
         for (name, desc, ty) in attr_commands {
             let name = name.strip_prefix(b"attr ").unwrap_or(&name);
@@ -62,14 +54,12 @@ impl Completer for AttributableCompletion {
         &mut self,
         working_set: &StateWorkingSet,
         _stack: &Stack,
-        _prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        let partial = working_set.get_span_contents(span);
-        let mut matcher = NuMatcher::new(String::from_utf8_lossy(partial), options.clone());
+        let mut matcher = NuMatcher::new(prefix, options);
 
         for s in ["def", "extern", "export def", "export extern"] {
             let decl_id = working_set
diff --git a/crates/nu-cli/src/completions/base.rs b/crates/nu-cli/src/completions/base.rs
index 359c2208b0f28..68956084954a3 100644
--- a/crates/nu-cli/src/completions/base.rs
+++ b/crates/nu-cli/src/completions/base.rs
@@ -12,10 +12,9 @@ pub trait Completer {
         &mut self,
         working_set: &StateWorkingSet,
         stack: &Stack,
-        prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion>;
 }
diff --git a/crates/nu-cli/src/completions/cell_path_completions.rs b/crates/nu-cli/src/completions/cell_path_completions.rs
index 3d921e8bbd088..7962d5be3f6c1 100644
--- a/crates/nu-cli/src/completions/cell_path_completions.rs
+++ b/crates/nu-cli/src/completions/cell_path_completions.rs
@@ -19,10 +19,9 @@ impl Completer for CellPathCompletion<'_> {
         &mut self,
         working_set: &StateWorkingSet,
         stack: &Stack,
-        _prefix: &[u8],
+        _prefix: impl AsRef<str>,
         _span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
         // empty tail is already handled as variable names completion
@@ -42,7 +41,7 @@ impl Completer for CellPathCompletion<'_> {
             end: true_end - offset,
         };
 
-        let mut matcher = NuMatcher::new(prefix_str, options.clone());
+        let mut matcher = NuMatcher::new(prefix_str, options);
 
         // evaluate the head expression to get its value
         let value = if let Expr::Var(var_id) = self.full_cell_path.head.expr {
diff --git a/crates/nu-cli/src/completions/command_completions.rs b/crates/nu-cli/src/completions/command_completions.rs
index ca54913ea1504..2f384840a1ce9 100644
--- a/crates/nu-cli/src/completions/command_completions.rs
+++ b/crates/nu-cli/src/completions/command_completions.rs
@@ -4,9 +4,8 @@ use crate::{
     completions::{Completer, CompletionOptions},
     SuggestionKind,
 };
-use nu_parser::FlatShape;
 use nu_protocol::{
-    engine::{CachedFile, Stack, StateWorkingSet},
+    engine::{Stack, StateWorkingSet},
     Span,
 };
 use reedline::Suggestion;
@@ -14,24 +13,13 @@ use reedline::Suggestion;
 use super::{completion_options::NuMatcher, SemanticSuggestion};
 
 pub struct CommandCompletion {
-    flattened: Vec<(Span, FlatShape)>,
-    flat_shape: FlatShape,
-    force_completion_after_space: bool,
+    /// Whether to include internal commands
+    pub internals: bool,
+    /// Whether to include external commands
+    pub externals: bool,
 }
 
 impl CommandCompletion {
-    pub fn new(
-        flattened: Vec<(Span, FlatShape)>,
-        flat_shape: FlatShape,
-        force_completion_after_space: bool,
-    ) -> Self {
-        Self {
-            flattened,
-            flat_shape,
-            force_completion_after_space,
-        }
-    }
-
     fn external_command_completion(
         &self,
         working_set: &StateWorkingSet,
@@ -71,6 +59,9 @@ impl CommandCompletion {
                             if suggs.contains_key(&value) {
                                 continue;
                             }
+                            // TODO: check name matching before a relative heavy IO involved
+                            // `is_executable` for performance consideration, should avoid
+                            // duplicated `match_aux` call for matched items in the future
                             if matcher.matches(&name) && is_executable::is_executable(item.path()) {
                                 // If there's an internal command with the same name, adds ^cmd to the
                                 // matcher so that both the internal and external command are included
@@ -97,46 +88,50 @@ impl CommandCompletion {
 
         suggs
     }
+}
 
-    fn complete_commands(
-        &self,
+impl Completer for CommandCompletion {
+    fn fetch(
+        &mut self,
         working_set: &StateWorkingSet,
+        _stack: &Stack,
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        find_externals: bool,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        let partial = working_set.get_span_contents(span);
-        let mut matcher = NuMatcher::new(String::from_utf8_lossy(partial), options.clone());
+        let mut matcher = NuMatcher::new(prefix, options);
 
         let sugg_span = reedline::Span::new(span.start - offset, span.end - offset);
 
         let mut internal_suggs = HashMap::new();
-        let filtered_commands = working_set.find_commands_by_predicate(
-            |name| {
-                let name = String::from_utf8_lossy(name);
-                matcher.add(&name, name.to_string())
-            },
-            true,
-        );
-        for (name, description, typ) in filtered_commands {
-            let name = String::from_utf8_lossy(&name);
-            internal_suggs.insert(
-                name.to_string(),
-                SemanticSuggestion {
-                    suggestion: Suggestion {
-                        value: name.to_string(),
-                        description,
-                        span: sugg_span,
-                        append_whitespace: true,
-                        ..Suggestion::default()
-                    },
-                    kind: Some(SuggestionKind::Command(typ)),
+        if self.internals {
+            let filtered_commands = working_set.find_commands_by_predicate(
+                |name| {
+                    let name = String::from_utf8_lossy(name);
+                    matcher.add(&name, name.to_string())
                 },
+                true,
             );
+            for (name, description, typ) in filtered_commands {
+                let name = String::from_utf8_lossy(&name);
+                internal_suggs.insert(
+                    name.to_string(),
+                    SemanticSuggestion {
+                        suggestion: Suggestion {
+                            value: name.to_string(),
+                            description,
+                            span: sugg_span,
+                            append_whitespace: true,
+                            ..Suggestion::default()
+                        },
+                        kind: Some(SuggestionKind::Command(typ)),
+                    },
+                );
+            }
         }
 
-        let mut external_suggs = if find_externals {
+        let mut external_suggs = if self.externals {
             self.external_command_completion(
                 working_set,
                 sugg_span,
@@ -159,179 +154,3 @@ impl CommandCompletion {
         res
     }
 }
-
-impl Completer for CommandCompletion {
-    fn fetch(
-        &mut self,
-        working_set: &StateWorkingSet,
-        _stack: &Stack,
-        _prefix: &[u8],
-        span: Span,
-        offset: usize,
-        pos: usize,
-        options: &CompletionOptions,
-    ) -> Vec<SemanticSuggestion> {
-        let last = self
-            .flattened
-            .iter()
-            .rev()
-            .skip_while(|x| x.0.end > pos)
-            .take_while(|x| {
-                matches!(
-                    x.1,
-                    FlatShape::InternalCall(_)
-                        | FlatShape::External
-                        | FlatShape::ExternalArg
-                        | FlatShape::Literal
-                        | FlatShape::String
-                )
-            })
-            .last();
-
-        // The last item here would be the earliest shape that could possible by part of this subcommand
-        let subcommands = if let Some(last) = last {
-            self.complete_commands(
-                working_set,
-                Span::new(last.0.start, pos),
-                offset,
-                false,
-                options,
-            )
-        } else {
-            vec![]
-        };
-
-        if !subcommands.is_empty() {
-            return subcommands;
-        }
-
-        let config = working_set.get_config();
-        if matches!(self.flat_shape, nu_parser::FlatShape::External)
-            || matches!(self.flat_shape, nu_parser::FlatShape::InternalCall(_))
-            || ((span.end - span.start) == 0)
-            || is_passthrough_command(working_set.delta.get_file_contents())
-        {
-            // we're in a gap or at a command
-            if working_set.get_span_contents(span).is_empty() && !self.force_completion_after_space
-            {
-                return vec![];
-            }
-            self.complete_commands(
-                working_set,
-                span,
-                offset,
-                config.completions.external.enable,
-                options,
-            )
-        } else {
-            vec![]
-        }
-    }
-}
-
-pub fn find_non_whitespace_index(contents: &[u8], start: usize) -> usize {
-    match contents.get(start..) {
-        Some(contents) => {
-            contents
-                .iter()
-                .take_while(|x| x.is_ascii_whitespace())
-                .count()
-                + start
-        }
-        None => start,
-    }
-}
-
-pub fn is_passthrough_command(working_set_file_contents: &[CachedFile]) -> bool {
-    for cached_file in working_set_file_contents {
-        let contents = &cached_file.content;
-        let last_pipe_pos_rev = contents.iter().rev().position(|x| x == &b'|');
-        let last_pipe_pos = last_pipe_pos_rev.map(|x| contents.len() - x).unwrap_or(0);
-
-        let cur_pos = find_non_whitespace_index(contents, last_pipe_pos);
-
-        let result = match contents.get(cur_pos..) {
-            Some(contents) => contents.starts_with(b"sudo ") || contents.starts_with(b"doas "),
-            None => false,
-        };
-        if result {
-            return true;
-        }
-    }
-    false
-}
-
-#[cfg(test)]
-mod command_completions_tests {
-    use super::*;
-    use nu_protocol::engine::EngineState;
-    use std::sync::Arc;
-
-    #[test]
-    fn test_find_non_whitespace_index() {
-        let commands = [
-            ("    hello", 4),
-            ("sudo ", 0),
-            (" 	sudo ", 2),
-            ("	 sudo ", 2),
-            ("	hello ", 1),
-            ("	  hello ", 3),
-            ("    hello | sudo ", 4),
-            ("     sudo|sudo", 5),
-            ("sudo | sudo ", 0),
-            ("	hello sud", 1),
-        ];
-        for (idx, ele) in commands.iter().enumerate() {
-            let index = find_non_whitespace_index(ele.0.as_bytes(), 0);
-            assert_eq!(index, ele.1, "Failed on index {}", idx);
-        }
-    }
-
-    #[test]
-    fn test_is_last_command_passthrough() {
-        let commands = [
-            ("    hello", false),
-            ("    sudo ", true),
-            ("sudo ", true),
-            ("	hello", false),
-            ("	sudo", false),
-            ("	sudo ", true),
-            (" 	sudo ", true),
-            ("	 sudo ", true),
-            ("	hello ", false),
-            ("    hello | sudo ", true),
-            ("    sudo|sudo", false),
-            ("sudo | sudo ", true),
-            ("	hello sud", false),
-            ("	sudo | sud ", false),
-            ("	sudo|sudo ", true),
-            (" 	sudo | sudo ls | sudo ", true),
-        ];
-        for (idx, ele) in commands.iter().enumerate() {
-            let input = ele.0.as_bytes();
-
-            let mut engine_state = EngineState::new();
-            engine_state.add_file("test.nu".into(), Arc::new([]));
-
-            let delta = {
-                let mut working_set = StateWorkingSet::new(&engine_state);
-                let _ = working_set.add_file("child.nu".into(), input);
-                working_set.render()
-            };
-
-            let result = engine_state.merge_delta(delta);
-            assert!(
-                result.is_ok(),
-                "Merge delta has failed: {}",
-                result.err().unwrap()
-            );
-
-            let is_passthrough_command = is_passthrough_command(engine_state.get_file_contents());
-            assert_eq!(
-                is_passthrough_command, ele.1,
-                "index for '{}': {}",
-                ele.0, idx
-            );
-        }
-    }
-}
diff --git a/crates/nu-cli/src/completions/completer.rs b/crates/nu-cli/src/completions/completer.rs
index dc89df1906e6d..457c19554f721 100644
--- a/crates/nu-cli/src/completions/completer.rs
+++ b/crates/nu-cli/src/completions/completer.rs
@@ -3,12 +3,11 @@ use crate::completions::{
     CompletionOptions, CustomCompletion, DirectoryCompletion, DotNuCompletion, FileCompletion,
     FlagCompletion, OperatorCompletion, VariableCompletion,
 };
-use log::debug;
 use nu_color_config::{color_record_to_nustyle, lookup_ansi_color_style};
 use nu_engine::eval_block;
-use nu_parser::{flatten_expression, parse, FlatShape};
+use nu_parser::{flatten_expression, parse};
 use nu_protocol::{
-    ast::{Expr, Expression, FindMapResult, Traverse},
+    ast::{Argument, Expr, Expression, FindMapResult, Traverse},
     debugger::WithoutDebug,
     engine::{Closure, EngineState, Stack, StateWorkingSet},
     PipelineData, Span, Value,
@@ -22,7 +21,7 @@ use super::base::{SemanticSuggestion, SuggestionKind};
 ///
 /// returns the inner-most pipeline_element of interest
 /// i.e. the one that contains given position and needs completion
-fn find_pipeline_element_by_position<'a>(
+pub fn find_pipeline_element_by_position<'a>(
     expr: &'a Expression,
     working_set: &'a StateWorkingSet,
     pos: usize,
@@ -41,7 +40,6 @@ fn find_pipeline_element_by_position<'a>(
             .or(Some(expr))
             .map(FindMapResult::Found)
             .unwrap_or_default(),
-        // TODO: clear separation of internal/external completion logic
         Expr::ExternalCall(head, arguments) => arguments
             .iter()
             .find_map(|arg| arg.expr().find_map(working_set, &closure))
@@ -85,12 +83,57 @@ fn strip_placeholder<'a>(working_set: &'a StateWorkingSet, span: &Span) -> (Span
     (new_span, prefix)
 }
 
+/// Given a span with noise,
+/// 1. Call `rsplit` to get the last token
+/// 2. Strip the last placeholder from the token
+fn strip_placeholder_with_rsplit<'a>(
+    working_set: &'a StateWorkingSet,
+    span: &Span,
+    predicate: impl FnMut(&u8) -> bool,
+) -> (Span, &'a [u8]) {
+    let span_content = working_set.get_span_contents(*span);
+    let mut prefix = span_content
+        .rsplit(predicate)
+        .next()
+        .unwrap_or(span_content);
+    let start = span.end.saturating_sub(prefix.len());
+    if !prefix.is_empty() {
+        prefix = &prefix[..prefix.len() - 1];
+    }
+    let end = start + prefix.len();
+    (Span::new(start, end), prefix)
+}
+
 #[derive(Clone)]
 pub struct NuCompleter {
     engine_state: Arc<EngineState>,
     stack: Stack,
 }
 
+/// Common arguments required for Completer
+struct Context<'a> {
+    working_set: &'a StateWorkingSet<'a>,
+    span: Span,
+    prefix: &'a [u8],
+    offset: usize,
+}
+
+impl Context<'_> {
+    fn new<'a>(
+        working_set: &'a StateWorkingSet,
+        span: Span,
+        prefix: &'a [u8],
+        offset: usize,
+    ) -> Context<'a> {
+        Context {
+            working_set,
+            span,
+            prefix,
+            offset,
+        }
+    }
+}
+
 impl NuCompleter {
     pub fn new(engine_state: Arc<EngineState>, stack: Arc<Stack>) -> Self {
         Self {
@@ -100,7 +143,245 @@ impl NuCompleter {
     }
 
     pub fn fetch_completions_at(&mut self, line: &str, pos: usize) -> Vec<SemanticSuggestion> {
-        self.completion_helper(line, pos)
+        let mut working_set = StateWorkingSet::new(&self.engine_state);
+        let offset = working_set.next_span_start();
+        // TODO: Callers should be trimming the line themselves
+        let line = if line.len() > pos { &line[..pos] } else { line };
+        // Adjust offset so that the spans of the suggestions will start at the right
+        // place even with `only_buffer_difference: true`
+        let pos = offset + pos;
+
+        let block = parse(
+            &mut working_set,
+            Some("completer"),
+            // Add a placeholder `a` to the end
+            format!("{}a", line).as_bytes(),
+            false,
+        );
+        let Some(element_expression) = block.find_map(&working_set, &|expr: &Expression| {
+            find_pipeline_element_by_position(expr, &working_set, pos)
+        }) else {
+            return vec![];
+        };
+
+        self.complete_by_expression(&working_set, element_expression, offset, pos, line)
+    }
+
+    /// Complete given the expression of interest
+    /// Usually, the expression is get from `find_pipeline_element_by_position`
+    ///
+    /// # Arguments
+    /// * `offset` - start offset of current working_set span
+    /// * `pos` - cursor position, should be > offset
+    /// * `prefix_str` - all the text before the cursor
+    pub fn complete_by_expression(
+        &self,
+        working_set: &StateWorkingSet,
+        element_expression: &Expression,
+        offset: usize,
+        pos: usize,
+        prefix_str: &str,
+    ) -> Vec<SemanticSuggestion> {
+        let mut suggestions: Vec<SemanticSuggestion> = vec![];
+
+        match &element_expression.expr {
+            Expr::Var(_) => {
+                return self.variable_names_completion_helper(
+                    working_set,
+                    element_expression.span,
+                    offset,
+                );
+            }
+            Expr::FullCellPath(full_cell_path) => {
+                // e.g. `$e<tab>` parsed as FullCellPath
+                if full_cell_path.tail.is_empty() {
+                    return self.variable_names_completion_helper(
+                        working_set,
+                        element_expression.span,
+                        offset,
+                    );
+                } else {
+                    let mut cell_path_completer = CellPathCompletion { full_cell_path };
+                    let ctx = Context::new(working_set, Span::unknown(), &[], offset);
+                    return self.process_completion(&mut cell_path_completer, &ctx);
+                }
+            }
+            Expr::BinaryOp(lhs, op, _) => {
+                if op.span.contains(pos) {
+                    let mut operator_completions = OperatorCompletion {
+                        left_hand_side: lhs.as_ref(),
+                    };
+                    let (new_span, prefix) = strip_placeholder(working_set, &op.span);
+                    let ctx = Context::new(working_set, new_span, prefix, offset);
+                    let results = self.process_completion(&mut operator_completions, &ctx);
+                    if !results.is_empty() {
+                        return results;
+                    }
+                }
+            }
+            Expr::AttributeBlock(ab) => {
+                if let Some(span) = ab.attributes.iter().find_map(|attr| {
+                    let span = attr.expr.span;
+                    span.contains(pos).then_some(span)
+                }) {
+                    let (new_span, prefix) = strip_placeholder(working_set, &span);
+                    let ctx = Context::new(working_set, new_span, prefix, offset);
+                    return self.process_completion(&mut AttributeCompletion, &ctx);
+                };
+                let span = ab.item.span;
+                if span.contains(pos) {
+                    let (new_span, prefix) = strip_placeholder(working_set, &span);
+                    let ctx = Context::new(working_set, new_span, prefix, offset);
+                    return self.process_completion(&mut AttributableCompletion, &ctx);
+                }
+            }
+
+            // NOTE: user defined internal commands can have any length
+            // e.g. `def "foo -f --ff bar"`, complete by line text
+            // instead of relying on the parsing result in that case
+            Expr::Call(_) | Expr::ExternalCall(_, _) => {
+                let need_externals = !prefix_str.contains(' ');
+                let need_internals = !prefix_str.starts_with('^');
+                let mut span = element_expression.span;
+                if !need_internals {
+                    span = Span::new(span.start + 1, span.end)
+                };
+                suggestions.extend(self.command_completion_helper(
+                    working_set,
+                    span,
+                    offset,
+                    need_internals,
+                    need_externals,
+                ))
+            }
+            _ => (),
+        }
+
+        // unfinished argument completion for commands
+        match &element_expression.expr {
+            Expr::Call(call) => {
+                // TODO: the argument to complete won't necessarily be the last one in the future
+                // for lsp completion, we won't trim the text,
+                // so that `def`s after pos can be completed
+                for arg in call.arguments.iter() {
+                    let span = arg.span();
+                    if span.contains(pos) {
+                        // if customized completion specified, it has highest priority
+                        if let Some(decl_id) = arg.expr().and_then(|e| e.custom_completion) {
+                            // for `--foo <tab>` and `--foo=<tab>`, the arg span should be trimmed
+                            let (new_span, prefix) = if matches!(arg, Argument::Named(_)) {
+                                strip_placeholder_with_rsplit(working_set, &span, |b| {
+                                    *b == b'=' || *b == b' '
+                                })
+                            } else {
+                                strip_placeholder(working_set, &span)
+                            };
+                            let ctx = Context::new(working_set, new_span, prefix, offset);
+
+                            let mut completer = CustomCompletion::new(
+                                decl_id,
+                                prefix_str.into(),
+                                pos - offset,
+                                FileCompletion,
+                            );
+
+                            suggestions.extend(self.process_completion(&mut completer, &ctx));
+                            break;
+                        }
+
+                        // normal arguments completion
+                        let (new_span, prefix) = strip_placeholder(working_set, &span);
+                        let ctx = Context::new(working_set, new_span, prefix, offset);
+                        suggestions.extend(match arg {
+                            // flags
+                            Argument::Named(_) | Argument::Unknown(_)
+                                if prefix.starts_with(b"-") =>
+                            {
+                                let mut flag_completions = FlagCompletion {
+                                    decl_id: call.decl_id,
+                                };
+                                self.process_completion(&mut flag_completions, &ctx)
+                            }
+                            // complete according to expression type and command head
+                            Argument::Positional(expr) => {
+                                let command_head = working_set.get_span_contents(call.head);
+                                self.argument_completion_helper(
+                                    command_head,
+                                    expr,
+                                    &ctx,
+                                    suggestions.is_empty(),
+                                )
+                            }
+                            _ => vec![],
+                        });
+                        break;
+                    }
+                }
+            }
+            Expr::ExternalCall(head, arguments) => {
+                for (i, arg) in arguments.iter().enumerate() {
+                    let span = arg.expr().span;
+                    if span.contains(pos) {
+                        // e.g. `sudo l<tab>`
+                        // HACK: judge by index 0 is not accurate
+                        if i == 0 {
+                            let external_cmd = working_set.get_span_contents(head.span);
+                            if external_cmd == b"sudo" || external_cmd == b"doas" {
+                                let commands = self.command_completion_helper(
+                                    working_set,
+                                    span,
+                                    offset,
+                                    true,
+                                    true,
+                                );
+                                // flags of sudo/doas can still be completed by external completer
+                                if !commands.is_empty() {
+                                    return commands;
+                                }
+                            }
+                        }
+                        // resort to external completer set in config
+                        let config = self.engine_state.get_config();
+                        if let Some(closure) = config.completions.external.completer.as_ref() {
+                            let mut text_spans: Vec<String> =
+                                flatten_expression(working_set, element_expression)
+                                    .iter()
+                                    .map(|(span, _)| {
+                                        let bytes = working_set.get_span_contents(*span);
+                                        String::from_utf8_lossy(bytes).to_string()
+                                    })
+                                    .collect();
+                            // strip the placeholder
+                            if let Some(last) = text_spans.last_mut() {
+                                last.pop();
+                            }
+                            if let Some(external_result) = self.external_completion(
+                                closure,
+                                &text_spans,
+                                offset,
+                                Span::new(span.start, span.end.saturating_sub(1)),
+                            ) {
+                                suggestions.extend(external_result);
+                                return suggestions;
+                            }
+                        }
+                        break;
+                    }
+                }
+            }
+            _ => (),
+        }
+
+        // if no suggestions yet, fallback to file completion
+        if suggestions.is_empty() {
+            let (new_span, prefix) =
+                strip_placeholder_with_rsplit(working_set, &element_expression.span, |c| {
+                    *c == b' '
+                });
+            let ctx = Context::new(working_set, new_span, prefix, offset);
+            suggestions.extend(self.process_completion(&mut FileCompletion, &ctx));
+        }
+        suggestions
     }
 
     fn variable_names_completion_helper(
@@ -113,27 +394,68 @@ impl NuCompleter {
         if !prefix.starts_with(b"$") {
             return vec![];
         }
-        let mut variable_names_completer = VariableCompletion {};
-        self.process_completion(
-            &mut variable_names_completer,
-            working_set,
-            prefix,
-            new_span,
-            offset,
-            // pos is not required
-            0,
-        )
+        let ctx = Context::new(working_set, new_span, prefix, offset);
+        self.process_completion(&mut VariableCompletion, &ctx)
+    }
+
+    fn command_completion_helper(
+        &self,
+        working_set: &StateWorkingSet,
+        span: Span,
+        offset: usize,
+        internals: bool,
+        externals: bool,
+    ) -> Vec<SemanticSuggestion> {
+        let mut command_completions = CommandCompletion {
+            internals,
+            externals,
+        };
+        let (new_span, prefix) = strip_placeholder(working_set, &span);
+        let ctx = Context::new(working_set, new_span, prefix, offset);
+        self.process_completion(&mut command_completions, &ctx)
+    }
+
+    fn argument_completion_helper(
+        &self,
+        command_head: &[u8],
+        expr: &Expression,
+        ctx: &Context,
+        need_fallback: bool,
+    ) -> Vec<SemanticSuggestion> {
+        // special commands
+        match command_head {
+            // complete module file/directory
+            // TODO: if module file already specified,
+            // should parse it to get modules/commands/consts to complete
+            b"use" | b"export use" | b"overlay use" | b"source-env" => {
+                return self.process_completion(&mut DotNuCompletion, ctx);
+            }
+            b"which" => {
+                let mut completer = CommandCompletion {
+                    internals: true,
+                    externals: true,
+                };
+                return self.process_completion(&mut completer, ctx);
+            }
+            _ => (),
+        }
+
+        // general positional arguments
+        let file_completion_helper = || self.process_completion(&mut FileCompletion, ctx);
+        match &expr.expr {
+            Expr::Directory(_, _) => self.process_completion(&mut DirectoryCompletion, ctx),
+            Expr::Filepath(_, _) | Expr::GlobPattern(_, _) => file_completion_helper(),
+            // fallback to file completion if necessary
+            _ if need_fallback => file_completion_helper(),
+            _ => vec![],
+        }
     }
 
     // Process the completion for a given completer
     fn process_completion<T: Completer>(
         &self,
         completer: &mut T,
-        working_set: &StateWorkingSet,
-        prefix: &[u8],
-        new_span: Span,
-        offset: usize,
-        pos: usize,
+        ctx: &Context,
     ) -> Vec<SemanticSuggestion> {
         let config = self.engine_state.get_config();
 
@@ -144,18 +466,12 @@ impl NuCompleter {
             ..Default::default()
         };
 
-        debug!(
-            "process_completion: prefix: {}, new_span: {new_span:?}, offset: {offset}, pos: {pos}",
-            String::from_utf8_lossy(prefix)
-        );
-
         completer.fetch(
-            working_set,
+            ctx.working_set,
             &self.stack,
-            prefix,
-            new_span,
-            offset,
-            pos,
+            String::from_utf8_lossy(ctx.prefix),
+            ctx.span,
+            ctx.offset,
             &options,
         )
     }
@@ -215,325 +531,11 @@ impl NuCompleter {
             }
         }
     }
-
-    fn completion_helper(&mut self, line: &str, pos: usize) -> Vec<SemanticSuggestion> {
-        let mut working_set = StateWorkingSet::new(&self.engine_state);
-        let offset = working_set.next_span_start();
-        // TODO: Callers should be trimming the line themselves
-        let line = if line.len() > pos { &line[..pos] } else { line };
-        // Adjust offset so that the spans of the suggestions will start at the right
-        // place even with `only_buffer_difference: true`
-        let fake_offset = offset + line.len() - pos;
-        let pos = offset + line.len();
-        let initial_line = line.to_string();
-        let mut line = line.to_string();
-        line.push('a');
-
-        let config = self.engine_state.get_config();
-
-        let block = parse(&mut working_set, Some("completer"), line.as_bytes(), false);
-        let Some(element_expression) = block.find_map(&working_set, &|expr: &Expression| {
-            find_pipeline_element_by_position(expr, &working_set, pos)
-        }) else {
-            return vec![];
-        };
-
-        match &element_expression.expr {
-            Expr::Var(_) => {
-                return self.variable_names_completion_helper(
-                    &working_set,
-                    element_expression.span,
-                    fake_offset,
-                );
-            }
-            Expr::FullCellPath(full_cell_path) => {
-                // e.g. `$e<tab>` parsed as FullCellPath
-                if full_cell_path.tail.is_empty() {
-                    return self.variable_names_completion_helper(
-                        &working_set,
-                        element_expression.span,
-                        fake_offset,
-                    );
-                } else {
-                    let mut cell_path_completer = CellPathCompletion { full_cell_path };
-                    return self.process_completion(
-                        &mut cell_path_completer,
-                        &working_set,
-                        &[],
-                        element_expression.span,
-                        fake_offset,
-                        pos,
-                    );
-                }
-            }
-            _ => (),
-        }
-
-        let flattened = flatten_expression(&working_set, element_expression);
-        let mut spans: Vec<String> = vec![];
-
-        for (flat_idx, (span, shape)) in flattened.iter().enumerate() {
-            let is_passthrough_command = spans
-                .first()
-                .filter(|content| content.as_str() == "sudo" || content.as_str() == "doas")
-                .is_some();
-
-            // Read the current span to string
-            let current_span = working_set.get_span_contents(*span);
-            let current_span_str = String::from_utf8_lossy(current_span);
-            let is_last_span = span.contains(pos);
-
-            // Skip the last 'a' as span item
-            if is_last_span {
-                let offset = pos - span.start;
-                if offset == 0 {
-                    spans.push(String::new())
-                } else {
-                    let mut current_span_str = current_span_str.to_string();
-                    current_span_str.remove(offset);
-                    spans.push(current_span_str);
-                }
-            } else {
-                spans.push(current_span_str.to_string());
-            }
-
-            // Complete based on the last span
-            if is_last_span {
-                // Create a new span
-                let new_span = Span::new(span.start, span.end - 1);
-
-                // Parses the prefix. Completion should look up to the cursor position, not after.
-                let index = pos - span.start;
-                let prefix = &current_span[..index];
-
-                if let Expr::AttributeBlock(ab) = &element_expression.expr {
-                    let last_attr = ab.attributes.last().expect("at least one attribute");
-                    if let Expr::Garbage = last_attr.expr.expr {
-                        return self.process_completion(
-                            &mut AttributeCompletion,
-                            &working_set,
-                            prefix,
-                            new_span,
-                            fake_offset,
-                            pos,
-                        );
-                    } else {
-                        return self.process_completion(
-                            &mut AttributableCompletion,
-                            &working_set,
-                            prefix,
-                            new_span,
-                            fake_offset,
-                            pos,
-                        );
-                    }
-                }
-
-                // Flags completion
-                if prefix.starts_with(b"-") {
-                    // Try to complete flag internally
-                    let mut completer = FlagCompletion::new(element_expression.clone());
-                    let result = self.process_completion(
-                        &mut completer,
-                        &working_set,
-                        prefix,
-                        new_span,
-                        fake_offset,
-                        pos,
-                    );
-
-                    if !result.is_empty() {
-                        return result;
-                    }
-
-                    // We got no results for internal completion
-                    // now we can check if external completer is set and use it
-                    if let Some(closure) = config.completions.external.completer.as_ref() {
-                        if let Some(external_result) =
-                            self.external_completion(closure, &spans, fake_offset, new_span)
-                        {
-                            return external_result;
-                        }
-                    }
-                }
-
-                // specially check if it is currently empty - always complete commands
-                if (is_passthrough_command && flat_idx == 1)
-                    || (flat_idx == 0 && working_set.get_span_contents(new_span).is_empty())
-                {
-                    let mut completer = CommandCompletion::new(
-                        flattened.clone(),
-                        // flat_idx,
-                        FlatShape::String,
-                        true,
-                    );
-                    return self.process_completion(
-                        &mut completer,
-                        &working_set,
-                        prefix,
-                        new_span,
-                        fake_offset,
-                        pos,
-                    );
-                }
-
-                // Completions that depends on the previous expression (e.g: use, source-env)
-                if (is_passthrough_command && flat_idx > 1) || flat_idx > 0 {
-                    if let Some(previous_expr) = flattened.get(flat_idx - 1) {
-                        // Read the content for the previous expression
-                        let prev_expr_str = working_set.get_span_contents(previous_expr.0).to_vec();
-
-                        // Completion for .nu files
-                        if prev_expr_str == b"use"
-                            || prev_expr_str == b"overlay use"
-                            || prev_expr_str == b"source-env"
-                        {
-                            let mut completer = DotNuCompletion::new();
-
-                            return self.process_completion(
-                                &mut completer,
-                                &working_set,
-                                prefix,
-                                new_span,
-                                fake_offset,
-                                pos,
-                            );
-                        } else if prev_expr_str == b"ls" {
-                            let mut completer = FileCompletion::new();
-
-                            return self.process_completion(
-                                &mut completer,
-                                &working_set,
-                                prefix,
-                                new_span,
-                                fake_offset,
-                                pos,
-                            );
-                        } else if matches!(
-                            previous_expr.1,
-                            FlatShape::Float
-                                | FlatShape::Int
-                                | FlatShape::String
-                                | FlatShape::List
-                                | FlatShape::Bool
-                                | FlatShape::Variable(_)
-                        ) {
-                            let mut completer = OperatorCompletion::new(element_expression.clone());
-
-                            let operator_suggestion = self.process_completion(
-                                &mut completer,
-                                &working_set,
-                                prefix,
-                                new_span,
-                                fake_offset,
-                                pos,
-                            );
-                            if !operator_suggestion.is_empty() {
-                                return operator_suggestion;
-                            }
-                        }
-                    }
-                }
-
-                // Match other types
-                match shape {
-                    FlatShape::Custom(decl_id) => {
-                        let mut completer = CustomCompletion::new(
-                            self.stack.clone(),
-                            *decl_id,
-                            initial_line,
-                            FileCompletion::new(),
-                        );
-
-                        return self.process_completion(
-                            &mut completer,
-                            &working_set,
-                            prefix,
-                            new_span,
-                            fake_offset,
-                            pos,
-                        );
-                    }
-                    FlatShape::Directory => {
-                        let mut completer = DirectoryCompletion::new();
-
-                        return self.process_completion(
-                            &mut completer,
-                            &working_set,
-                            prefix,
-                            new_span,
-                            fake_offset,
-                            pos,
-                        );
-                    }
-                    FlatShape::Filepath | FlatShape::GlobPattern => {
-                        let mut completer = FileCompletion::new();
-
-                        return self.process_completion(
-                            &mut completer,
-                            &working_set,
-                            prefix,
-                            new_span,
-                            fake_offset,
-                            pos,
-                        );
-                    }
-                    flat_shape => {
-                        let mut completer = CommandCompletion::new(
-                            flattened.clone(),
-                            // flat_idx,
-                            flat_shape.clone(),
-                            false,
-                        );
-
-                        let mut out: Vec<_> = self.process_completion(
-                            &mut completer,
-                            &working_set,
-                            prefix,
-                            new_span,
-                            fake_offset,
-                            pos,
-                        );
-
-                        if !out.is_empty() {
-                            return out;
-                        }
-
-                        // Try to complete using an external completer (if set)
-                        if let Some(closure) = config.completions.external.completer.as_ref() {
-                            if let Some(external_result) =
-                                self.external_completion(closure, &spans, fake_offset, new_span)
-                            {
-                                return external_result;
-                            }
-                        }
-
-                        // Check for file completion
-                        let mut completer = FileCompletion::new();
-                        out = self.process_completion(
-                            &mut completer,
-                            &working_set,
-                            prefix,
-                            new_span,
-                            fake_offset,
-                            pos,
-                        );
-
-                        if !out.is_empty() {
-                            return out;
-                        }
-                    }
-                };
-            }
-        }
-
-        vec![]
-    }
 }
 
 impl ReedlineCompleter for NuCompleter {
     fn complete(&mut self, line: &str, pos: usize) -> Vec<Suggestion> {
-        self.completion_helper(line, pos)
+        self.fetch_completions_at(line, pos)
             .into_iter()
             .map(|s| s.suggestion)
             .collect()
@@ -656,7 +658,7 @@ mod completer_tests {
             ("ls | sudo m", true, "m", vec!["mv", "mut", "move"]),
         ];
         for (line, has_result, begins_with, expected_values) in dataset {
-            let result = completer.completion_helper(line, line.len());
+            let result = completer.fetch_completions_at(line, line.len());
             // Test whether the result is empty or not
             assert_eq!(!result.is_empty(), has_result, "line: {}", line);
 
diff --git a/crates/nu-cli/src/completions/completion_common.rs b/crates/nu-cli/src/completions/completion_common.rs
index 0a060666b878a..91b83381c3d85 100644
--- a/crates/nu-cli/src/completions/completion_common.rs
+++ b/crates/nu-cli/src/completions/completion_common.rs
@@ -51,7 +51,7 @@ fn complete_rec(
     }
 
     let prefix = partial.first().unwrap_or(&"");
-    let mut matcher = NuMatcher::new(prefix, options.clone());
+    let mut matcher = NuMatcher::new(prefix, options);
 
     for built in built_paths {
         let mut path = built.cwd.clone();
@@ -315,12 +315,12 @@ pub struct AdjustView {
 }
 
 pub fn adjust_if_intermediate(
-    prefix: &[u8],
+    prefix: &str,
     working_set: &StateWorkingSet,
     mut span: nu_protocol::Span,
 ) -> AdjustView {
     let span_contents = String::from_utf8_lossy(working_set.get_span_contents(span)).to_string();
-    let mut prefix = String::from_utf8_lossy(prefix).to_string();
+    let mut prefix = prefix.to_string();
 
     // A difference of 1 because of the cursor's unicode code point in between.
     // Using .chars().count() because unicode and Windows.
diff --git a/crates/nu-cli/src/completions/completion_options.rs b/crates/nu-cli/src/completions/completion_options.rs
index cbe119e2c1742..1d33ea26a90a4 100644
--- a/crates/nu-cli/src/completions/completion_options.rs
+++ b/crates/nu-cli/src/completions/completion_options.rs
@@ -25,8 +25,8 @@ pub enum MatchAlgorithm {
     Fuzzy,
 }
 
-pub struct NuMatcher<T> {
-    options: CompletionOptions,
+pub struct NuMatcher<'a, T> {
+    options: &'a CompletionOptions,
     needle: String,
     state: State<T>,
 }
@@ -45,11 +45,11 @@ enum State<T> {
 }
 
 /// Filters and sorts suggestions
-impl<T> NuMatcher<T> {
+impl<T> NuMatcher<'_, T> {
     /// # Arguments
     ///
     /// * `needle` - The text to search for
-    pub fn new(needle: impl AsRef<str>, options: CompletionOptions) -> NuMatcher<T> {
+    pub fn new(needle: impl AsRef<str>, options: &CompletionOptions) -> NuMatcher<T> {
         let needle = trim_quotes_str(needle.as_ref());
         match options.match_algorithm {
             MatchAlgorithm::Prefix => {
@@ -184,7 +184,7 @@ impl<T> NuMatcher<T> {
     }
 }
 
-impl NuMatcher<SemanticSuggestion> {
+impl NuMatcher<'_, SemanticSuggestion> {
     pub fn add_semantic_suggestion(&mut self, sugg: SemanticSuggestion) -> bool {
         let value = sugg.suggestion.value.to_string();
         self.add(value, sugg)
@@ -271,7 +271,7 @@ mod test {
             match_algorithm,
             ..Default::default()
         };
-        let mut matcher = NuMatcher::new(needle, options);
+        let mut matcher = NuMatcher::new(needle, &options);
         matcher.add(haystack, haystack);
         if should_match {
             assert_eq!(vec![haystack], matcher.results());
@@ -286,7 +286,7 @@ mod test {
             match_algorithm: MatchAlgorithm::Fuzzy,
             ..Default::default()
         };
-        let mut matcher = NuMatcher::new("fob", options);
+        let mut matcher = NuMatcher::new("fob", &options);
         for item in ["foo/bar", "fob", "foo bar"] {
             matcher.add(item, item);
         }
@@ -300,7 +300,7 @@ mod test {
             match_algorithm: MatchAlgorithm::Fuzzy,
             ..Default::default()
         };
-        let mut matcher = NuMatcher::new("'love spaces' ", options);
+        let mut matcher = NuMatcher::new("'love spaces' ", &options);
         for item in [
             "'i love spaces'",
             "'i love spaces' so much",
diff --git a/crates/nu-cli/src/completions/custom_completions.rs b/crates/nu-cli/src/completions/custom_completions.rs
index 852ea130f8bf0..317c865de1126 100644
--- a/crates/nu-cli/src/completions/custom_completions.rs
+++ b/crates/nu-cli/src/completions/custom_completions.rs
@@ -13,18 +13,18 @@ use std::collections::HashMap;
 use super::completion_options::NuMatcher;
 
 pub struct CustomCompletion<T: Completer> {
-    stack: Stack,
     decl_id: DeclId,
     line: String,
+    line_pos: usize,
     fallback: T,
 }
 
 impl<T: Completer> CustomCompletion<T> {
-    pub fn new(stack: Stack, decl_id: DeclId, line: String, fallback: T) -> Self {
+    pub fn new(decl_id: DeclId, line: String, line_pos: usize, fallback: T) -> Self {
         Self {
-            stack,
             decl_id,
             line,
+            line_pos,
             fallback,
         }
     }
@@ -35,19 +35,16 @@ impl<T: Completer> Completer for CustomCompletion<T> {
         &mut self,
         working_set: &StateWorkingSet,
         stack: &Stack,
-        prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        pos: usize,
         orig_options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        // Line position
-        let line_pos = pos - offset;
-
         // Call custom declaration
+        let mut stack_mut = stack.clone();
         let result = eval_call::<WithoutDebug>(
             working_set.permanent_state,
-            &mut self.stack,
+            &mut stack_mut,
             &Call {
                 decl_id: self.decl_id,
                 head: span,
@@ -58,7 +55,7 @@ impl<T: Completer> Completer for CustomCompletion<T> {
                         Type::String,
                     )),
                     Argument::Positional(Expression::new_unknown(
-                        Expr::Int(line_pos as i64),
+                        Expr::Int(self.line_pos as i64),
                         Span::unknown(),
                         Type::Int,
                     )),
@@ -120,7 +117,6 @@ impl<T: Completer> Completer for CustomCompletion<T> {
                         prefix,
                         span,
                         offset,
-                        pos,
                         orig_options,
                     );
                 }
@@ -138,7 +134,7 @@ impl<T: Completer> Completer for CustomCompletion<T> {
             }
         };
 
-        let mut matcher = NuMatcher::new(String::from_utf8_lossy(prefix), completion_options);
+        let mut matcher = NuMatcher::new(prefix, &completion_options);
 
         if should_sort {
             for sugg in suggestions {
diff --git a/crates/nu-cli/src/completions/directory_completions.rs b/crates/nu-cli/src/completions/directory_completions.rs
index b23b45945f767..78639b7e7a47b 100644
--- a/crates/nu-cli/src/completions/directory_completions.rs
+++ b/crates/nu-cli/src/completions/directory_completions.rs
@@ -11,27 +11,20 @@ use std::path::Path;
 
 use super::{completion_common::FileSuggestion, SemanticSuggestion};
 
-#[derive(Clone, Default)]
-pub struct DirectoryCompletion {}
-
-impl DirectoryCompletion {
-    pub fn new() -> Self {
-        Self::default()
-    }
-}
+pub struct DirectoryCompletion;
 
 impl Completer for DirectoryCompletion {
     fn fetch(
         &mut self,
         working_set: &StateWorkingSet,
         stack: &Stack,
-        prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        let AdjustView { prefix, span, .. } = adjust_if_intermediate(prefix, working_set, span);
+        let AdjustView { prefix, span, .. } =
+            adjust_if_intermediate(prefix.as_ref(), working_set, span);
 
         // Filter only the folders
         #[allow(deprecated)]
diff --git a/crates/nu-cli/src/completions/dotnu_completions.rs b/crates/nu-cli/src/completions/dotnu_completions.rs
index 7bb3458fe04c9..084d52d65bcab 100644
--- a/crates/nu-cli/src/completions/dotnu_completions.rs
+++ b/crates/nu-cli/src/completions/dotnu_completions.rs
@@ -12,27 +12,19 @@ use std::{
 
 use super::{SemanticSuggestion, SuggestionKind};
 
-#[derive(Clone, Default)]
-pub struct DotNuCompletion {}
-
-impl DotNuCompletion {
-    pub fn new() -> Self {
-        Self::default()
-    }
-}
+pub struct DotNuCompletion;
 
 impl Completer for DotNuCompletion {
     fn fetch(
         &mut self,
         working_set: &StateWorkingSet,
         stack: &Stack,
-        prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        let prefix_str = String::from_utf8_lossy(prefix);
+        let prefix_str = prefix.as_ref();
         let start_with_backquote = prefix_str.starts_with('`');
         let end_with_backquote = prefix_str.ends_with('`');
         let prefix_str = prefix_str.replace('`', "");
diff --git a/crates/nu-cli/src/completions/file_completions.rs b/crates/nu-cli/src/completions/file_completions.rs
index 5b4c5468782de..07993c78117d9 100644
--- a/crates/nu-cli/src/completions/file_completions.rs
+++ b/crates/nu-cli/src/completions/file_completions.rs
@@ -11,31 +11,23 @@ use std::path::Path;
 
 use super::{completion_common::FileSuggestion, SemanticSuggestion};
 
-#[derive(Clone, Default)]
-pub struct FileCompletion {}
-
-impl FileCompletion {
-    pub fn new() -> Self {
-        Self::default()
-    }
-}
+pub struct FileCompletion;
 
 impl Completer for FileCompletion {
     fn fetch(
         &mut self,
         working_set: &StateWorkingSet,
         stack: &Stack,
-        prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
         let AdjustView {
             prefix,
             span,
             readjusted,
-        } = adjust_if_intermediate(prefix, working_set, span);
+        } = adjust_if_intermediate(prefix.as_ref(), working_set, span);
 
         #[allow(deprecated)]
         let items: Vec<_> = complete_item(
diff --git a/crates/nu-cli/src/completions/flag_completions.rs b/crates/nu-cli/src/completions/flag_completions.rs
index 1df95f04bad40..c07ee93c9b941 100644
--- a/crates/nu-cli/src/completions/flag_completions.rs
+++ b/crates/nu-cli/src/completions/flag_completions.rs
@@ -1,8 +1,7 @@
 use crate::completions::{completion_options::NuMatcher, Completer, CompletionOptions};
 use nu_protocol::{
-    ast::{Expr, Expression},
     engine::{Stack, StateWorkingSet},
-    Span,
+    DeclId, Span,
 };
 use reedline::Suggestion;
 
@@ -10,13 +9,7 @@ use super::SemanticSuggestion;
 
 #[derive(Clone)]
 pub struct FlagCompletion {
-    expression: Expression,
-}
-
-impl FlagCompletion {
-    pub fn new(expression: Expression) -> Self {
-        Self { expression }
-    }
+    pub decl_id: DeclId,
 }
 
 impl Completer for FlagCompletion {
@@ -24,69 +17,43 @@ impl Completer for FlagCompletion {
         &mut self,
         working_set: &StateWorkingSet,
         _stack: &Stack,
-        prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        // Check if it's a flag
-        if let Expr::Call(call) = &self.expression.expr {
-            let decl = working_set.get_decl(call.decl_id);
-            let sig = decl.signature();
-
-            let mut matcher = NuMatcher::new(String::from_utf8_lossy(prefix), options.clone());
-
-            for named in &sig.named {
-                let flag_desc = &named.desc;
-                if let Some(short) = named.short {
-                    let mut named = vec![0; short.len_utf8()];
-                    short.encode_utf8(&mut named);
-                    named.insert(0, b'-');
-
-                    matcher.add_semantic_suggestion(SemanticSuggestion {
-                        suggestion: Suggestion {
-                            value: String::from_utf8_lossy(&named).to_string(),
-                            description: Some(flag_desc.to_string()),
-                            span: reedline::Span {
-                                start: span.start - offset,
-                                end: span.end - offset,
-                            },
-                            append_whitespace: true,
-                            ..Suggestion::default()
-                        },
-                        // TODO????
-                        kind: None,
-                    });
-                }
-
-                if named.long.is_empty() {
-                    continue;
-                }
-
-                let mut named = named.long.as_bytes().to_vec();
-                named.insert(0, b'-');
-                named.insert(0, b'-');
-
-                matcher.add_semantic_suggestion(SemanticSuggestion {
-                    suggestion: Suggestion {
-                        value: String::from_utf8_lossy(&named).to_string(),
-                        description: Some(flag_desc.to_string()),
-                        span: reedline::Span {
-                            start: span.start - offset,
-                            end: span.end - offset,
-                        },
-                        append_whitespace: true,
-                        ..Suggestion::default()
+        let mut matcher = NuMatcher::new(prefix, options);
+        let mut add_suggestion = |value: String, description: String| {
+            matcher.add_semantic_suggestion(SemanticSuggestion {
+                suggestion: Suggestion {
+                    value,
+                    description: Some(description),
+                    span: reedline::Span {
+                        start: span.start - offset,
+                        end: span.end - offset,
                     },
-                    // TODO????
-                    kind: None,
-                });
+                    append_whitespace: true,
+                    ..Suggestion::default()
+                },
+                // TODO????
+                kind: None,
+            });
+        };
+
+        let decl = working_set.get_decl(self.decl_id);
+        let sig = decl.signature();
+        for named in &sig.named {
+            if let Some(short) = named.short {
+                let mut name = String::from("-");
+                name.push(short);
+                add_suggestion(name, named.desc.clone());
             }
 
-            return matcher.results();
+            if named.long.is_empty() {
+                continue;
+            }
+            add_suggestion(format!("--{}", named.long), named.desc.clone());
         }
-
-        vec![]
+        matcher.results()
     }
 }
diff --git a/crates/nu-cli/src/completions/operator_completions.rs b/crates/nu-cli/src/completions/operator_completions.rs
index 0e5e1d9c909b9..31793e00e5674 100644
--- a/crates/nu-cli/src/completions/operator_completions.rs
+++ b/crates/nu-cli/src/completions/operator_completions.rs
@@ -9,36 +9,22 @@ use nu_protocol::{
 use reedline::Suggestion;
 
 #[derive(Clone)]
-pub struct OperatorCompletion {
-    previous_expr: Expression,
+pub struct OperatorCompletion<'a> {
+    pub left_hand_side: &'a Expression,
 }
 
-impl OperatorCompletion {
-    pub fn new(previous_expr: Expression) -> Self {
-        OperatorCompletion { previous_expr }
-    }
-}
-
-impl Completer for OperatorCompletion {
+impl Completer for OperatorCompletion<'_> {
     fn fetch(
         &mut self,
         working_set: &StateWorkingSet,
         _stack: &Stack,
-        _prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
         //Check if int, float, or string
-        let partial = std::str::from_utf8(working_set.get_span_contents(span)).unwrap_or("");
-        let op = match &self.previous_expr.expr {
-            Expr::BinaryOp(x, _, _) => &x.expr,
-            _ => {
-                return vec![];
-            }
-        };
-        let possible_operations = match op {
+        let possible_operations = match &self.left_hand_side.expr {
             Expr::Int(_) => vec![
                 ("+", "Add (Plus)"),
                 ("-", "Subtract (Minus)"),
@@ -121,7 +107,7 @@ impl Completer for OperatorCompletion {
             _ => vec![],
         };
 
-        let mut matcher = NuMatcher::new(partial, options.clone());
+        let mut matcher = NuMatcher::new(prefix, options);
         for (symbol, desc) in possible_operations.into_iter() {
             matcher.add_semantic_suggestion(SemanticSuggestion {
                 suggestion: Suggestion {
diff --git a/crates/nu-cli/src/completions/variable_completions.rs b/crates/nu-cli/src/completions/variable_completions.rs
index b251f451e463b..3e0fdebe79262 100644
--- a/crates/nu-cli/src/completions/variable_completions.rs
+++ b/crates/nu-cli/src/completions/variable_completions.rs
@@ -7,21 +7,19 @@ use reedline::Suggestion;
 
 use super::completion_options::NuMatcher;
 
-pub struct VariableCompletion {}
+pub struct VariableCompletion;
 
 impl Completer for VariableCompletion {
     fn fetch(
         &mut self,
         working_set: &StateWorkingSet,
         _stack: &Stack,
-        prefix: &[u8],
+        prefix: impl AsRef<str>,
         span: Span,
         offset: usize,
-        _pos: usize,
         options: &CompletionOptions,
     ) -> Vec<SemanticSuggestion> {
-        let prefix_str = String::from_utf8_lossy(prefix);
-        let mut matcher = NuMatcher::new(prefix_str, options.clone());
+        let mut matcher = NuMatcher::new(prefix, options);
         let current_span = reedline::Span {
             start: span.start - offset,
             end: span.end - offset,


Test patch:
diff --git a/crates/nu-cli/tests/completions/mod.rs b/crates/nu-cli/tests/completions/mod.rs
index d97916f2c9e4f..27fba186ea4a4 100644
--- a/crates/nu-cli/tests/completions/mod.rs
+++ b/crates/nu-cli/tests/completions/mod.rs
@@ -14,7 +14,9 @@ use nu_protocol::{debugger::WithoutDebug, engine::StateWorkingSet, PipelineData}
 use reedline::{Completer, Suggestion};
 use rstest::{fixture, rstest};
 use support::{
-    completions_helpers::{new_dotnu_engine, new_partial_engine, new_quote_engine},
+    completions_helpers::{
+        new_dotnu_engine, new_external_engine, new_partial_engine, new_quote_engine,
+    },
     file, folder, match_suggestions, new_engine,
 };
 
@@ -292,6 +294,105 @@ fn customcompletions_fallback() {
     match_suggestions(&expected, &suggestions);
 }
 
+/// Custom function arguments mixed with subcommands
+#[test]
+fn custom_arguments_and_subcommands() {
+    let (_, _, mut engine, mut stack) = new_engine();
+    let command = r#"
+        def foo [i: directory] {}
+        def "foo test bar" [] {}"#;
+    assert!(support::merge_input(command.as_bytes(), &mut engine, &mut stack).is_ok());
+
+    let mut completer = NuCompleter::new(Arc::new(engine), Arc::new(stack));
+    let completion_str = "foo test";
+    let suggestions = completer.complete(completion_str, completion_str.len());
+    // including both subcommand and directory completions
+    let expected: Vec<String> = vec!["foo test bar".into(), folder("test_a"), folder("test_b")];
+    match_suggestions(&expected, &suggestions);
+}
+
+/// Custom function flags mixed with subcommands
+#[test]
+fn custom_flags_and_subcommands() {
+    let (_, _, mut engine, mut stack) = new_engine();
+    let command = r#"
+        def foo [--test: directory] {}
+        def "foo --test bar" [] {}"#;
+    assert!(support::merge_input(command.as_bytes(), &mut engine, &mut stack).is_ok());
+
+    let mut completer = NuCompleter::new(Arc::new(engine), Arc::new(stack));
+    let completion_str = "foo --test";
+    let suggestions = completer.complete(completion_str, completion_str.len());
+    // including both flag and directory completions
+    let expected: Vec<String> = vec!["foo --test bar".into(), "--test".into()];
+    match_suggestions(&expected, &suggestions);
+}
+
+/// If argument type is something like int/string, complete only subcommands
+#[test]
+fn custom_arguments_vs_subcommands() {
+    let (_, _, mut engine, mut stack) = new_engine();
+    let command = r#"
+        def foo [i: string] {}
+        def "foo test bar" [] {}"#;
+    assert!(support::merge_input(command.as_bytes(), &mut engine, &mut stack).is_ok());
+
+    let mut completer = NuCompleter::new(Arc::new(engine), Arc::new(stack));
+    let completion_str = "foo test";
+    let suggestions = completer.complete(completion_str, completion_str.len());
+    // including only subcommand completions
+    let expected: Vec<String> = vec!["foo test bar".into()];
+    match_suggestions(&expected, &suggestions);
+}
+
+/// External command only if starts with `^`
+#[test]
+fn external_commands_only() {
+    let engine = new_external_engine();
+    let mut completer = NuCompleter::new(
+        Arc::new(engine),
+        Arc::new(nu_protocol::engine::Stack::new()),
+    );
+    let completion_str = "^sleep";
+    let suggestions = completer.complete(completion_str, completion_str.len());
+    #[cfg(windows)]
+    let expected: Vec<String> = vec!["sleep.exe".into()];
+    #[cfg(not(windows))]
+    let expected: Vec<String> = vec!["sleep".into()];
+    match_suggestions(&expected, &suggestions);
+
+    let completion_str = "sleep";
+    let suggestions = completer.complete(completion_str, completion_str.len());
+    #[cfg(windows)]
+    let expected: Vec<String> = vec!["sleep".into(), "sleep.exe".into()];
+    #[cfg(not(windows))]
+    let expected: Vec<String> = vec!["sleep".into(), "^sleep".into()];
+    match_suggestions(&expected, &suggestions);
+}
+
+/// Which completes both internals and externals
+#[test]
+fn which_command_completions() {
+    let engine = new_external_engine();
+    let mut completer = NuCompleter::new(
+        Arc::new(engine),
+        Arc::new(nu_protocol::engine::Stack::new()),
+    );
+    // flags
+    let completion_str = "which --all";
+    let suggestions = completer.complete(completion_str, completion_str.len());
+    let expected: Vec<String> = vec!["--all".into()];
+    match_suggestions(&expected, &suggestions);
+    // commands
+    let completion_str = "which sleep";
+    let suggestions = completer.complete(completion_str, completion_str.len());
+    #[cfg(windows)]
+    let expected: Vec<String> = vec!["sleep".into(), "sleep.exe".into()];
+    #[cfg(not(windows))]
+    let expected: Vec<String> = vec!["sleep".into(), "^sleep".into()];
+    match_suggestions(&expected, &suggestions);
+}
+
 /// Suppress completions for invalid values
 #[test]
 fn customcompletions_invalid() {
@@ -307,6 +408,25 @@ fn customcompletions_invalid() {
     assert!(suggestions.is_empty());
 }
 
+#[test]
+fn dont_use_dotnu_completions() {
+    // Create a new engine
+    let (_, _, engine, stack) = new_dotnu_engine();
+    // Instantiate a new completer
+    let mut completer = NuCompleter::new(Arc::new(engine), Arc::new(stack));
+    // Test nested nu script
+    let completion_str = "go work use `./dir_module/".to_string();
+    let suggestions = completer.complete(&completion_str, completion_str.len());
+
+    // including a plaintext file
+    let expected: Vec<String> = vec![
+        "./dir_module/mod.nu".into(),
+        "./dir_module/plain.txt".into(),
+        "`./dir_module/sub module/`".into(),
+    ];
+    match_suggestions(&expected, &suggestions);
+}
+
 #[test]
 fn dotnu_completions() {
     // Create a new engine
@@ -315,6 +435,15 @@ fn dotnu_completions() {
     // Instantiate a new completer
     let mut completer = NuCompleter::new(Arc::new(engine), Arc::new(stack));
 
+    // Flags should still be working
+    let completion_str = "overlay use --".to_string();
+    let suggestions = completer.complete(&completion_str, completion_str.len());
+
+    match_suggestions(
+        &vec!["--help".into(), "--prefix".into(), "--reload".into()],
+        &suggestions,
+    );
+
     // Test nested nu script
     #[cfg(windows)]
     let completion_str = "use `.\\dir_module\\".to_string();
@@ -486,6 +615,17 @@ fn external_completer_fallback() {
     match_suggestions(&expected, &suggestions);
 }
 
+/// Fallback to external completions for flags of `sudo`
+#[test]
+fn external_completer_sudo() {
+    let block = "{|spans| ['--background']}";
+    let input = "sudo --back".to_string();
+
+    let expected = vec!["--background".into()];
+    let suggestions = run_external_completion(block, &input);
+    match_suggestions(&expected, &suggestions);
+}
+
 /// Suppress completions when external completer returns invalid value
 #[test]
 fn external_completer_invalid() {
diff --git a/crates/nu-cli/tests/completions/support/completions_helpers.rs b/crates/nu-cli/tests/completions/support/completions_helpers.rs
index ed541997a3a61..c04397d019289 100644
--- a/crates/nu-cli/tests/completions/support/completions_helpers.rs
+++ b/crates/nu-cli/tests/completions/support/completions_helpers.rs
@@ -14,7 +14,7 @@ fn create_default_context() -> EngineState {
     nu_command::add_shell_command_context(nu_cmd_lang::create_default_context())
 }
 
-// creates a new engine with the current path into the completions fixtures folder
+/// creates a new engine with the current path into the completions fixtures folder
 pub fn new_engine() -> (AbsolutePathBuf, String, EngineState, Stack) {
     // Target folder inside assets
     let dir = fs::fixtures().join("completions");
@@ -69,7 +69,26 @@ pub fn new_engine() -> (AbsolutePathBuf, String, EngineState, Stack) {
     (dir, dir_str, engine_state, stack)
 }
 
-// creates a new engine with the current path into the completions fixtures folder
+/// Adds pseudo PATH env for external completion tests
+pub fn new_external_engine() -> EngineState {
+    let mut engine = create_default_context();
+    let dir = fs::fixtures().join("external_completions").join("path");
+    let dir_str = dir.to_string_lossy().to_string();
+    let internal_span = nu_protocol::Span::new(0, dir_str.len());
+    engine.add_env_var(
+        "PATH".to_string(),
+        Value::List {
+            vals: vec![Value::String {
+                val: dir_str,
+                internal_span,
+            }],
+            internal_span,
+        },
+    );
+    engine
+}
+
+/// creates a new engine with the current path into the completions fixtures folder
 pub fn new_dotnu_engine() -> (AbsolutePathBuf, String, EngineState, Stack) {
     // Target folder inside assets
     let dir = fs::fixtures().join("dotnu_completions");
@@ -197,7 +216,7 @@ pub fn new_partial_engine() -> (AbsolutePathBuf, String, EngineState, Stack) {
     (dir, dir_str, engine_state, stack)
 }
 
-// match a list of suggestions with the expected values
+/// match a list of suggestions with the expected values
 pub fn match_suggestions(expected: &Vec<String>, suggestions: &Vec<Suggestion>) {
     let expected_len = expected.len();
     let suggestions_len = suggestions.len();
@@ -209,28 +228,28 @@ pub fn match_suggestions(expected: &Vec<String>, suggestions: &Vec<Suggestion>)
         )
     }
 
-    let suggestoins_str = suggestions
+    let suggestions_str = suggestions
         .iter()
         .map(|it| it.value.clone())
         .collect::<Vec<_>>();
 
-    assert_eq!(expected, &suggestoins_str);
+    assert_eq!(expected, &suggestions_str);
 }
 
-// append the separator to the converted path
+/// append the separator to the converted path
 pub fn folder(path: impl Into<PathBuf>) -> String {
     let mut converted_path = file(path);
     converted_path.push(MAIN_SEPARATOR);
     converted_path
 }
 
-// convert a given path to string
+/// convert a given path to string
 pub fn file(path: impl Into<PathBuf>) -> String {
     path.into().into_os_string().into_string().unwrap()
 }
 
-// merge_input executes the given input into the engine
-// and merges the state
+/// merge_input executes the given input into the engine
+/// and merges the state
 pub fn merge_input(
     input: &[u8],
     engine_state: &mut EngineState,
diff --git a/tests/fixtures/dotnu_completions/dir_module/plain.txt b/tests/fixtures/dotnu_completions/dir_module/plain.txt
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/fixtures/external_completions/path/sleep b/tests/fixtures/external_completions/path/sleep
new file mode 100755
index 0000000000000..e69de29bb2d1d
diff --git a/tests/fixtures/external_completions/path/sleep.exe b/tests/fixtures/external_completions/path/sleep.exe
new file mode 100644
index 0000000000000..e69de29bb2d1d