Repository: nats-io/natscli
Pull request #1356: Support sending on newline for `nats pub`
Author: login: thewizzy
Created: 2025-04-26T02:52:30Z | Merged: 2025-04-30T09:56:31Z

Closes #1345 

The general purpose is to allow programs to pipe into natscli and have it send a message for each newline. 

* Implements `--send-on=newline`, also supports `eof`` which is considered the default behavior. 
* Implements SendOn as a type so that comparisons were quicker.
* Implements a signal.NotifyContext to terminate the program more sanely. Similar pattern to other places.
* Implemented `--quiet`/`-q` for pub to minimize non error (INFO) related logging. Not a show stopped but difficult to implement this globally without overhauling logging. 

Tested for both normal and JetStream publishing. 

Linked issue #1345: Support sending on newline for `nats pub`
### Proposed change

Provide a CLI flag for `nats pub <topic>` to tell it to send a message on every newline.

Example of sending on newline:

```sh
cat requests.jsonl | nats pub request --send-on-newline
```

Or if instead you wanted it more flexible for future use cases:

```sh
# What I would like it to do
cat requests.jsonl | nats pub request --send-on=newline
# The default and what it does today
cat requests.jsonl | nats pub request --send-on=eof
```

-----

Note: If this feature already exists or is possible somewhere, my apologies for the request. The idea is based off another's feedback + me skimming the docs and help options to see if something was missed. I didn't see an option there.

### Use case

I have a tool that can listen on stdin and publish on stdout in JSONL format.

It would be nice to be able to do:

```sh
nats sub request | my-command | nats pub response --send-on=newline
```

Or even for simple transforms like:

```sh
nats sub raw | jq -cr 'some-transform' | nats pub formatted --send-on=newline
```


### Contribution

Yep! If the proposal is accepted, I'd be up for me (or someone on my team) submitting a PR with the things needed to get the feature added ^_^
Linked issue comments: user: login: ripienaar
created_at: 2025-04-18T11:43:55Z
body: I think it's a good idea, would be happy to review.  Would need to be careful and support it for both JetStream and non JetStream publishing.
author_association: COLLABORATOR
user: login: bplaxco
created_at: 2025-04-22T22:06:41Z
body: >  Would need to be careful and support it for both JetStream and non JetStream publishing.

Okay, thanks for that note. ^_^
author_association: NONE
user: login: alexbozhenko
created_at: 2025-04-29T21:17:28Z
body: Does it have to be part of the natscli?

I feel like one can achieve that with [xargs](https://www.man7.org/linux/man-pages/man1/xargs.1.html)?
```
cat /tmp/jsonl.txt | xargs -I % nats pub subject "%"
```

It has this flag:
```
       --delimiter=delim, -d delim
              Input items are terminated by the specified character. ...
```
That would allow split input on any char.

author_association: MEMBER
user: login: thewizzy
created_at: 2025-04-29T23:05:59Z
body: > Does it have to be part of the natscli?
> 
> I feel like one can achieve that with [xargs](https://www.man7.org/linux/man-pages/man1/xargs.1.html)?
> 
> ```
> cat /tmp/jsonl.txt | xargs -I % nats pub subject "%"
> ```
> 
> It has this flag:
> 
> ```
>        --delimiter=delim, -d delim
>               Input items are terminated by the specified character. ...
> ```
> 
> That would allow split input on any char.

It is correct, this could be achieved with xargs, though it could catch delimiters (like newlines \n) in the body. 

More significantly is that every line of the file executes`nats`, connects to the server and then cleans up the process. 

I did a simple test to see how significant the different was, the server was located on the same local network but a different machine.
**xargs - using release 0.2.2**
`time cat nats/tests/consumer_command_test.go | xargs -I % -d \n nats pub testing -s 192.168.1.2 "%"`
```
real    0m30.511s
user    0m5.517s
sys     0m4.088s
```
Note: to be fair the delimiter here does mean that the xargs method published more lines.

**send-on newline**
`time cat nats/tests/consumer_command_test.go |  nats pub testing -s 192.168.1.2 --send-on newline`
```
real    0m0.814s
user    0m0.032s
sys     0m0.114s
```
author_association: CONTRIBUTOR
user: login: ripienaar
created_at: 2025-05-16T09:21:16Z
body: Done in https://github.com/nats-io/natscli/pull/1356
author_association: COLLABORATOR

Diff:
diff --git a/cli/cheats/pub.md b/cli/cheats/pub.md
index 65b10384..1d8f4a80 100644
--- a/cli/cheats/pub.md
+++ b/cli/cheats/pub.md
@@ -9,3 +9,6 @@ echo "hello world" | nats pub --force-stdin destination.subject
 
 # To request a response from a server and show just the raw result
 nats request destination.subject "hello world" -H "Content-type:text/plain" --raw
+
+# To listen on STDIN and publish one message per newline
+nats pub destination.subject --send-on=newline
\ No newline at end of file
diff --git a/cli/pub_command.go b/cli/pub_command.go
index d688fec9..db82afca 100644
--- a/cli/pub_command.go
+++ b/cli/pub_command.go
@@ -14,10 +14,14 @@
 package cli
 
 import (
+	"bufio"
+	"bytes"
 	"fmt"
 	"io"
 	"math"
 	"os"
+	"os/signal"
+	"syscall"
 	"time"
 
 	iu "github.com/nats-io/natscli/internal/util"
@@ -29,6 +33,11 @@ import (
 	terminal "golang.org/x/term"
 )
 
+const (
+	sendOnEOF     = "eof"
+	sendOnNewline = "newline"
+)
+
 type pubCmd struct {
 	subject      string
 	body         string
@@ -44,6 +53,8 @@ type pubCmd struct {
 	forceStdin   bool
 	translate    string
 	jetstream    bool
+	sendOn       string
+	quiet        bool
 }
 
 func configurePubCommand(app commandHost) {
@@ -80,6 +91,9 @@ Available template functions are:
 	pub.Flag("sleep", "When publishing multiple messages, sleep between publishes").DurationVar(&c.sleep)
 	pub.Flag("force-stdin", "Force reading from stdin").UnNegatableBoolVar(&c.forceStdin)
 	pub.Flag("jetstream", "Publish messages to jetstream").Short('J').UnNegatableBoolVar(&c.jetstream)
+	pub.Flag("send-on", fmt.Sprintf("When to send data from stdin: '%s' (default) or '%s'", sendOnEOF, sendOnNewline)).
+		Default("eof").EnumVar(&c.sendOn, sendOnNewline, sendOnEOF)
+	pub.Flag("quiet", "Show just the output received").Short('q').UnNegatableBoolVar(&c.quiet)
 
 	requestHelp := `Body and Header values of the messages may use Go templates to 
 create unique messages.
@@ -204,7 +218,7 @@ func (c *pubCmd) doReq(nc *nats.Conn, progress *progress.Tracker) error {
 							log.Printf("%s: %s", h, val)
 						}
 					}
-					fmt.Println()
+					log.Println()
 				}
 
 				outPutMSGBody(m.Data, c.translate, m.Subject, "")
@@ -259,9 +273,9 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 		}
 
 		msg.Subject = string(subj)
-
-		log.Printf("Published %d bytes to %q\n", len(body), c.subject)
-
+		if !c.quiet {
+			log.Printf("Published %d bytes to %q\n", len(body), c.subject)
+		}
 		resp, err := nc.RequestMsg(msg, opts().Timeout)
 		if err != nil {
 			return err
@@ -278,7 +292,7 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 
 		if progress != nil {
 			progress.Increment(1)
-		} else {
+		} else if !c.quiet {
 			msg := fmt.Sprintf("Stored in Stream: %s Sequence: %s", ack.Stream, f(ack.Sequence))
 			if ack.Domain != "" {
 				msg += fmt.Sprintf(" Domain: %q", ack.Domain)
@@ -301,88 +315,180 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 	return nil
 }
 
+// readLine reads a full line from a bufio.Reader regardless of buffer size,
+// this is important for reading lines longer than the bufio.Reader's default
+// buffer size and nicer than wholesale enlarging the buffer.
+func readLine(reader *bufio.Reader) (string, error) {
+	var buf bytes.Buffer
+
+	for {
+		line, isPrefix, err := reader.ReadLine()
+		if err != nil {
+			return buf.String(), err
+		}
+
+		buf.Write(line)
+		if !isPrefix {
+			return buf.String(), err
+		}
+	}
+}
+
 func (c *pubCmd) publish(_ *fisk.ParseContext) error {
+	ctx, cancel := signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT, syscall.SIGQUIT)
+	defer cancel()
+
 	nc, err := newNatsConn("", natsOpts()...)
 	if err != nil {
 		return err
 	}
 	defer nc.Close()
 
-	if c.cnt < 1 {
-		c.cnt = math.MaxInt16
-	}
+	// Create a pipe so that we can interrupt the input at any time.
+	readPipe, writePipe := io.Pipe()
+	go func() {
+		_, errPipe := io.Copy(writePipe, os.Stdin)
+		if errPipe != nil {
+			writePipe.CloseWithError(errPipe)
+		} else {
+			// io.Copy does not return io.EOF, so on success we need to trigger EOF that by closing the pipe
+			_ = writePipe.Close()
+		}
+	}()
+	reader := bufio.NewReader(readPipe)
+	complete := make(chan struct{})
 
-	if !c.bodyIsSet && (terminal.IsTerminal(int(os.Stdout.Fd())) || c.forceStdin) {
+	// If a body is set, treat it as EOF, as no more input
+	eof := c.bodyIsSet
+	if !c.bodyIsSet && !c.quiet && (terminal.IsTerminal(int(os.Stdout.Fd())) || c.forceStdin) {
 		log.Println("Reading payload from STDIN")
-		body, err := io.ReadAll(os.Stdin)
-		if err != nil {
-			return err
-		}
-		c.body = string(body)
 	}
 
-	var tracker *progress.Tracker
-	var progbar progress.Writer
+	errCh := make(chan error, 1)
+	go func() {
+		defer close(complete)
+		for {
+			if c.cnt < 1 {
+				c.cnt = math.MaxInt16
+			}
 
-	if c.cnt > 20 && !c.raw {
-		progbar, tracker, err = iu.NewProgress(opts(), &progress.Tracker{
-			Total: int64(c.cnt),
-		})
-		if err != nil {
-			return err
-		}
+			if !c.bodyIsSet && (terminal.IsTerminal(int(os.Stdout.Fd())) || c.forceStdin) {
+				if c.sendOn == sendOnEOF {
+					body, err := io.ReadAll(readPipe)
+					if err != nil {
+						errCh <- err
+						return
+					}
+					c.body = string(body)
+				} else if c.sendOn == sendOnNewline {
+					body, err := readLine(reader)
+					if err != nil && err != io.EOF {
+						errCh <- err
+						return
+					} else if err == io.EOF {
+						eof = true
+					}
+					if body == "" && eof {
+						errCh <- nil
+						return
+					}
+					c.body = body
+				}
+			}
 
-		defer func() {
-			progbar.Stop()
-			time.Sleep(300 * time.Millisecond)
-		}()
-	}
+			var tracker *progress.Tracker
+			var progbar progress.Writer
 
-	if c.jetstream {
-		return c.doJetstream(nc, tracker)
-	}
+			if c.cnt > 20 && !c.raw {
+				progbar, tracker, err = iu.NewProgress(opts(), &progress.Tracker{
+					Total: int64(c.cnt),
+				})
+				if err != nil {
+					errCh <- err
+					return
+				}
 
-	if c.req || c.replyCount >= 1 {
-		return c.doReq(nc, tracker)
-	}
+				defer func() {
+					progbar.Stop()
+					time.Sleep(300 * time.Millisecond)
+				}()
+			}
 
-	for i := 1; i <= c.cnt; i++ {
-		body, err := iu.PubReplyBodyTemplate(c.body, "", i)
-		if err != nil {
-			log.Printf("Could not parse body template: %s", err)
-		}
+			if c.jetstream {
+				err = c.doJetstream(nc, tracker)
+				if c.sendOn == sendOnEOF {
+					errCh <- err
+					return
+				} else if c.sendOn == sendOnNewline {
+					if err != nil {
+						log.Printf("Could not publish message: %s", err)
+					}
+					continue
+				}
+			}
 
-		subj, err := iu.PubReplyBodyTemplate(c.subject, "", i)
-		if err != nil {
-			log.Printf("Could not parse subject template: %s", err)
-		}
+			if c.req || c.replyCount >= 1 {
+				errCh <- c.doReq(nc, tracker)
+				return
+			}
 
-		msg, err := c.prepareMsg(string(subj), body, i)
-		if err != nil {
-			return err
-		}
+			for i := 1; i <= c.cnt; i++ {
+				body, err := iu.PubReplyBodyTemplate(c.body, "", i)
+				if err != nil {
+					log.Printf("Could not parse body template: %s", err)
+				}
 
-		err = nc.PublishMsg(msg)
-		if err != nil {
-			return err
-		}
-		nc.Flush()
+				subj, err := iu.PubReplyBodyTemplate(c.subject, "", i)
+				if err != nil {
+					log.Printf("Could not parse subject template: %s", err)
+				}
 
-		err = nc.LastError()
-		if err != nil {
-			return err
-		}
+				msg, err := c.prepareMsg(string(subj), body, i)
+				if err != nil {
+					errCh <- err
+					return
+				}
 
-		if c.cnt > 1 && c.sleep > 0 {
-			time.Sleep(c.sleep)
-		}
+				err = nc.PublishMsg(msg)
+				if err != nil {
+					errCh <- err
+					return
+				}
+				nc.Flush()
 
-		if progbar == nil {
-			log.Printf("Published %d bytes to %q\n", len(body), c.subject)
-		} else {
-			tracker.Increment(1)
+				err = nc.LastError()
+				if err != nil {
+					errCh <- err
+					return
+				}
+
+				if c.cnt > 1 && c.sleep > 0 {
+					time.Sleep(c.sleep)
+				}
+
+				if progbar == nil {
+					if !c.quiet {
+						log.Printf("Published %d bytes to %q\n", len(body), c.subject)
+					}
+				} else {
+					tracker.Increment(1)
+				}
+			}
+
+			if c.sendOn == sendOnEOF || eof {
+				errCh <- nil
+				return
+			}
 		}
+	}()
+
+	// Wait until the core go routine is complete or context is canceled.
+	// Closes the remote connection(due to defer), the core loop go routine is just dropped.
+	select {
+	case <-ctx.Done():
+		readPipe.Close()
+		return fmt.Errorf("interrupted")
+	case <-complete:
+		return <-errCh
 	}
-
-	return nil
 }
diff --git a/nats/tests/pub_command_test.go b/nats/tests/pub_command_test.go
new file mode 100644
index 00000000..7d17762e
--- /dev/null
+++ b/nats/tests/pub_command_test.go
@@ -0,0 +1,97 @@
+package main
+
+import (
+	"fmt"
+	"github.com/nats-io/nats.go"
+	"strings"
+	"testing"
+)
+
+func TestCLIPubSendOnNewline(t *testing.T) {
+	srv, _, _ := setupJStreamTest(t)
+	defer srv.Shutdown()
+	nc, _, _ := prepareHelper(srv.ClientURL())
+
+	t.Run("Publish with body argument", func(t *testing.T) {
+		subject := "test-body"
+		var messages []string
+		expected := "Test Message"
+		sub, _ := nc.Subscribe(subject, func(m *nats.Msg) {
+			messages = append(messages, string(m.Data))
+		})
+
+		runNatsCli(t, fmt.Sprintf("--server='%s' pub --send-on newline %s '%s'", srv.ClientURL(), subject, expected))
+		_ = sub.Unsubscribe()
+
+		if len(messages) != 1 {
+			t.Errorf("expected 1 message and received %d", len(messages))
+		}
+		if len(messages) > 0 && messages[0] != expected {
+			t.Errorf("expected message %q got %q", expected, messages[0])
+		}
+	})
+
+	t.Run("Publish with body argument and --quiet", func(t *testing.T) {
+		subject := "test-quiet"
+		var messages []string
+		expected := "Test Message"
+		sub, _ := nc.Subscribe(subject, func(m *nats.Msg) {
+			messages = append(messages, string(m.Data))
+		})
+
+		out := runNatsCli(t, fmt.Sprintf("--server='%s' pub --send-on newline -q %s '%s'", srv.ClientURL(), subject, expected))
+		_ = sub.Unsubscribe()
+
+		if len(out) != 0 {
+			t.Errorf("expected cli to output to be 0 but got %d\n %s", len(out), out)
+		}
+		if len(messages) != 1 {
+			t.Errorf("expected 1 message and received %d", len(messages))
+		}
+		if len(messages) > 0 && messages[0] != expected {
+			t.Errorf("expected message %q got %q", expected, messages[0])
+		}
+	})
+
+	t.Run("Publish --send-on newline from stdin", func(t *testing.T) {
+		subject := "test-sendon"
+		var messages []string
+		expected := []string{"test", "pub", "input"}
+		sub, _ := nc.Subscribe(subject, func(m *nats.Msg) {
+			messages = append(messages, string(m.Data))
+		})
+
+		// --force-stdin required for testing as a terminal is not present
+		runNatsCliWithInput(t, strings.Join(expected, "\n"), fmt.Sprintf("--server='%s' pub --send-on newline --force-stdin %s", srv.ClientURL(), subject))
+		_ = sub.Unsubscribe()
+
+		if len(messages) != len(expected) {
+			t.Errorf("expected %d messages and received %d", len(expected), len(messages))
+		}
+		for i, msg := range messages {
+			if messages[i] != expected[i] {
+				t.Errorf("expected message(%d) %q got %q", i, expected[i], msg)
+			}
+		}
+	})
+
+	t.Run("Publish --send-on eof from stdin", func(t *testing.T) {
+		subject := "test-eof"
+		var messages []string
+		expected := "test\npub\ninput"
+		sub, _ := nc.Subscribe(subject, func(m *nats.Msg) {
+			messages = append(messages, string(m.Data))
+		})
+
+		// --force-stdin required for testing as a terminal is not present
+		runNatsCliWithInput(t, expected, fmt.Sprintf("--server='%s' pub --force-stdin %s", srv.ClientURL(), subject))
+		_ = sub.Unsubscribe()
+
+		if len(messages) != 1 {
+			t.Errorf("expected 1 message and received %d", len(messages))
+		}
+		if len(messages) > 0 && messages[0] != expected {
+			t.Errorf("expected message %q got %q", expected, messages[0])
+		}
+	})
+}


Reviews:
user: login: ripienaar
body: Some basic comments - will do a proper look at the logic next week
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-04-26T10:53:21Z
user: login: thewizzy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T11:04:00Z
user: login: ripienaar
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-26T11:05:30Z
user: login: thewizzy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T11:17:59Z
user: login: thewizzy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T11:24:57Z
user: login: ripienaar
body: There seems to be quite a significant issue here, my understanding is that this should works:

```
$ echo "test\npub\ninput"|bin/nats pub --send-on newline x
```

Should send 3 messages and terminate.  Instead it sends just `test` followed by infinity of empty body messages in a tight loop

```
$ echo "test\npub\ninput"|nats pub --send-on newline x
$ nats sub x --count 5
08:28:33 Subscribing on x
[#1] Received on "x"
test


[#2] Received on "x"
nil body


[#3] Received on "x"
nil body


[#4] Received on "x"
nil body


[#5] Received on "x"
nil body
```
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T08:30:16Z
user: login: ripienaar
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-29T06:38:41Z
user: login: thewizzy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T02:48:29Z
user: login: ripienaar
body: LGTM
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-30T09:54:08Z

Review comments:
diff_hunk: @@ -80,6 +118,11 @@ Available template functions are:
 	pub.Flag("sleep", "When publishing multiple messages, sleep between publishes").DurationVar(&c.sleep)
 	pub.Flag("force-stdin", "Force reading from stdin").UnNegatableBoolVar(&c.forceStdin)
 	pub.Flag("jetstream", "Publish messages to jetstream").Short('J').UnNegatableBoolVar(&c.jetstream)
+	pub.Flag("send-on", "When to send data from stdin: 'eof' (default) or 'newline'").Default("eof").Action(func(v *fisk.ParseContext) error {
path: cli/pub_command.go
user: login: ripienaar
body: Think we dont need the whole type and String(), Set() etc - elsewhere we use a `string` and then the flag `EnumVar()` feature like this:

https://github.com/nats-io/natscli/blob/456256a702ede68bb24aef62fdd4cae1136cfc3f/cli/stream_command.go#L191

That works fine and is much easier.
created_at: 2025-04-26T10:49:31Z
author_association: COLLABORATOR
diff_hunk: @@ -301,88 +344,137 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 	return nil
 }
 
+// readLine reads a full line from a bufio.Reader regardless of buffer size,
+// this is important for reading lines longer than the bufio.Reader's default
+// buffer size and nicer than wholesale enlarging the buffer.
+func readLine(reader *bufio.Reader) (string, error) {
+	var buf bytes.Buffer
+
+	for {
+		line, isPrefix, err := reader.ReadLine()
path: cli/pub_command.go
user: login: ripienaar
body: Would a `buffio.Scanner()` be easier here?
created_at: 2025-04-26T10:51:04Z
author_association: COLLABORATOR
diff_hunk: @@ -301,88 +344,137 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 	return nil
 }
 
+// readLine reads a full line from a bufio.Reader regardless of buffer size,
+// this is important for reading lines longer than the bufio.Reader's default
+// buffer size and nicer than wholesale enlarging the buffer.
+func readLine(reader *bufio.Reader) (string, error) {
+	var buf bytes.Buffer
+
+	for {
+		line, isPrefix, err := reader.ReadLine()
+		buf.Write(line)
+
+		if err != nil || !isPrefix {
path: cli/pub_command.go
user: login: ripienaar
body: If the readline above fails would we want to Write() it into buff? Seems weird to have error handling after the write
created_at: 2025-04-26T10:51:56Z
author_association: COLLABORATOR
diff_hunk: @@ -80,6 +118,11 @@ Available template functions are:
 	pub.Flag("sleep", "When publishing multiple messages, sleep between publishes").DurationVar(&c.sleep)
 	pub.Flag("force-stdin", "Force reading from stdin").UnNegatableBoolVar(&c.forceStdin)
 	pub.Flag("jetstream", "Publish messages to jetstream").Short('J').UnNegatableBoolVar(&c.jetstream)
+	pub.Flag("send-on", "When to send data from stdin: 'eof' (default) or 'newline'").Default("eof").Action(func(v *fisk.ParseContext) error {
path: cli/pub_command.go
user: login: thewizzy
body: I initially had it this way. With string comparisons in a few places. 
Realistically, whilst the current way is a few orders of magnitude quicker, its probably insignificant in the overall scheme of each message. If strings are your preference, happy to change back.
created_at: 2025-04-26T11:04:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -80,6 +118,11 @@ Available template functions are:
 	pub.Flag("sleep", "When publishing multiple messages, sleep between publishes").DurationVar(&c.sleep)
 	pub.Flag("force-stdin", "Force reading from stdin").UnNegatableBoolVar(&c.forceStdin)
 	pub.Flag("jetstream", "Publish messages to jetstream").Short('J').UnNegatableBoolVar(&c.jetstream)
+	pub.Flag("send-on", "When to send data from stdin: 'eof' (default) or 'newline'").Default("eof").Action(func(v *fisk.ParseContext) error {
path: cli/pub_command.go
user: login: ripienaar
body: Yeah, it would be good to stick to established patterns I think 
created_at: 2025-04-26T11:05:30Z
author_association: COLLABORATOR
diff_hunk: @@ -301,88 +344,137 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 	return nil
 }
 
+// readLine reads a full line from a bufio.Reader regardless of buffer size,
+// this is important for reading lines longer than the bufio.Reader's default
+// buffer size and nicer than wholesale enlarging the buffer.
+func readLine(reader *bufio.Reader) (string, error) {
+	var buf bytes.Buffer
+
+	for {
+		line, isPrefix, err := reader.ReadLine()
+		buf.Write(line)
+
+		if err != nil || !isPrefix {
path: cli/pub_command.go
user: login: thewizzy
body: Agree it does seem odd. 

If `bufio.ReadlLine()` fails then `line` is `nil` and `buf.Write(nil)` which has a length of 0 and nothing happens. It does make the return path simpler.

I can rewrite it as the following or add a comment to explain the oddity?
```
	for {
		line, isPrefix, err := reader.ReadLine()

		if err != nil {
			return buf.String(), err
		}

		buf.Write(line)
		if !isPrefix {
			return buf.String(), err
		}
	}
```
created_at: 2025-04-26T11:17:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -301,88 +344,137 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 	return nil
 }
 
+// readLine reads a full line from a bufio.Reader regardless of buffer size,
+// this is important for reading lines longer than the bufio.Reader's default
+// buffer size and nicer than wholesale enlarging the buffer.
+func readLine(reader *bufio.Reader) (string, error) {
+	var buf bytes.Buffer
+
+	for {
+		line, isPrefix, err := reader.ReadLine()
path: cli/pub_command.go
user: login: thewizzy
body: If you have a line longer than 64KB you will get a `token too long` error and it will truncate the input (not great with a default max message of 1MB). We have run into this in a few places and this has been our go to solution. It is generally preferred over defaulting to a much larger buffer as this would be allocated for every case and most people will probably not hit this. 

`io.ReadAll()` does not experience this issue as it keeps reading and allocating memory. 
created_at: 2025-04-26T11:24:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,38 +33,11 @@ import (
 	terminal "golang.org/x/term"
 )
 
-type SendOn int
-
 const (
-	SendOnEOF SendOn = iota
-	SendOnNewline
+	SendOnEOF     = "eof"
path: cli/pub_command.go
user: login: ripienaar
body: these dont need to be exported
created_at: 2025-04-28T08:08:51Z
author_association: COLLABORATOR
diff_hunk: @@ -301,88 +344,137 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 	return nil
 }
 
+// readLine reads a full line from a bufio.Reader regardless of buffer size,
+// this is important for reading lines longer than the bufio.Reader's default
+// buffer size and nicer than wholesale enlarging the buffer.
+func readLine(reader *bufio.Reader) (string, error) {
+	var buf bytes.Buffer
+
+	for {
+		line, isPrefix, err := reader.ReadLine()
+		buf.Write(line)
+
+		if err != nil || !isPrefix {
path: cli/pub_command.go
user: login: ripienaar
body: yeah that seems a bit less surprising.  No blank line above error checking.
created_at: 2025-04-28T08:10:01Z
author_association: COLLABORATOR
diff_hunk: @@ -301,88 +315,137 @@ func (c *pubCmd) doJetstream(nc *nats.Conn, progress *progress.Tracker) error {
 	return nil
 }
 
+// readLine reads a full line from a bufio.Reader regardless of buffer size,
+// this is important for reading lines longer than the bufio.Reader's default
+// buffer size and nicer than wholesale enlarging the buffer.
+func readLine(reader *bufio.Reader) (string, error) {
+	var buf bytes.Buffer
+
+	for {
+		line, isPrefix, err := reader.ReadLine()
+		buf.Write(line)
+
+		if err != nil || !isPrefix {
+			return buf.String(), err
+		}
+	}
+}
+
 func (c *pubCmd) publish(_ *fisk.ParseContext) error {
+	ctx, cancel := signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)
+	defer cancel()
+
 	nc, err := newNatsConn("", natsOpts()...)
 	if err != nil {
 		return err
 	}
 	defer nc.Close()
-
-	if c.cnt < 1 {
-		c.cnt = math.MaxInt16
-	}
-
-	if !c.bodyIsSet && (terminal.IsTerminal(int(os.Stdout.Fd())) || c.forceStdin) {
+	if !c.quiet {
 		log.Println("Reading payload from STDIN")
-		body, err := io.ReadAll(os.Stdin)
-		if err != nil {
-			return err
-		}
-		c.body = string(body)
 	}
+	for {
+		select {
+		case <-ctx.Done():
+			return fmt.Errorf("interrupted")
+		default:
+			{
path: cli/pub_command.go
user: login: ripienaar
body: not sure adding a block here adds any value other than make the code more annoying
created_at: 2025-04-28T08:26:02Z
author_association: COLLABORATOR
diff_hunk: @@ -958,3 +958,47 @@ func TestCLIMessageRm(t *testing.T) {
 		t.Fatalf("loading delete message did not fail")
 	}
 }
+
+func TestCLIPubSendOnNewline(t *testing.T) {
path: nats/tests/nats_test.go
user: login: ripienaar
body: mind moving this to `pub_command_test.go` like `sub_command_test.go`?

These are probably ok but would be better to set up a sub using the nats connection listening for these messages and verifying we get the right messages
created_at: 2025-04-29T06:38:41Z
author_association: COLLABORATOR
diff_hunk: @@ -958,3 +958,47 @@ func TestCLIMessageRm(t *testing.T) {
 		t.Fatalf("loading delete message did not fail")
 	}
 }
+
+func TestCLIPubSendOnNewline(t *testing.T) {
path: nats/tests/nats_test.go
user: login: thewizzy
body: I have made these changes. I kept the cli output only when it was part of the test, `--quiet` mode.
created_at: 2025-04-30T02:48:29Z
author_association: CONTRIBUTOR

PR comments:
user: login: ripienaar
created_at: 2025-04-28T09:59:50Z
body: Getting nearer, the above test command never exits once EOF is reached. May be worth adding some tests into `nats/test` directory there are some examples.  You can focus just on this behaviour we will circle back and flesh it out with more tests
author_association: COLLABORATOR
user: login: ripienaar
created_at: 2025-04-28T10:04:44Z
body: Also just doing `nats pub --send-on newline x x` will again forever publish the same bytes
author_association: COLLABORATOR
user: login: thewizzy
created_at: 2025-04-28T10:45:01Z
body: Sorry if my push pulled you back in early. The current commit handles the cases you have shown. However, it will send an empty message on a line with no content, it will also send an empty message on SIGINT. Considering whether it is worthwhile having the cancellation since it cannot interrupt a readline, it also simplifies the code.

I appreciate you looking into this. 

> There seems to be quite a significant issue here, my understanding is that this should works:
> 
> ```
> $ echo "test\npub\ninput"|bin/nats pub --send-on newline x
> ```
Yes, I was not handling EOF properly and was creating a new reader each time(losing buffer). `bufio.ReadLine` does not evaluate literal `\n` in a string as a newline, only "real" newlines - and having it evaluate newlines in a string would likely introduce issues. 

The following will work as `-e` interprets the escapes creating "real" newlines.
```
$ echo -e "test\npub\ninput"|bin/nats pub --send-on newline x
```

author_association: CONTRIBUTOR
user: login: ripienaar
created_at: 2025-04-28T16:33:41Z
body: Looks good - we must have adjusted main since your fork and probably moved some tests around, if you fix those I think we can mark is as ready
author_association: COLLABORATOR
user: login: ripienaar
created_at: 2025-04-28T16:48:23Z
body: This `echo` vs `echo -e` thing had me confused - turns out `zsh` builtin echo defaults to `-e` while `/bin/echo` and maybe bash defaults to `-E` :)
author_association: COLLABORATOR
user: login: thewizzy
created_at: 2025-04-28T23:07:54Z
body: > Looks good - we must have adjusted main since your fork and probably moved some tests around, if you fix those I think we can mark is as ready

Thank you for looking into it. What are your thoughts on removing the signal handling? Because of the way `bufio.ReadLine` works we cannot interrupt it properly until we get a newline - could send an empty message. Not fussed either way.

Also, I have been testing on our project and we get a few errors like `template: body:1: function "lookup" not defined`. The issue being we will sometimes have `{{ template }}` in the messages. It would be good to implement another option `--no-templates` or similar. Would you prefer a separate PR for this?

Thank you again for your efforts reviewing my PR and feedback.
author_association: CONTRIBUTOR
user: login: ripienaar
created_at: 2025-04-29T06:34:12Z
body: Yeah I think a flag like that will be good, the way we do them is like this:

https://github.com/nats-io/natscli/blob/456256a702ede68bb24aef62fdd4cae1136cfc3f/cli/stream_command.go#L219

Ie we add the `--templates` option and default it to `true`, the system will then automatically add `--no-templates`.  You probably wouldn't need the `IsSetByUser()` part

Best in a new PR
author_association: COLLABORATOR
user: login: ploubser
created_at: 2025-04-29T12:22:59Z
body: I think we will need to find a way to handle SIGINT before we merge this. When a user runs `natscli pub destination.subject` with the proposed PR the client doesn't terminate with ctrl+c, forcing them to kill it manually to exit. 
author_association: COLLABORATOR
user: login: thewizzy
created_at: 2025-04-30T02:53:49Z
body: > I think we will need to find a way to handle SIGINT before we merge this. When a user runs `natscli pub destination.subject` with the proposed PR the client doesn't terminate with ctrl+c, forcing them to kill it manually to exit.

I have implemented better signal handling that handles the IO differently. We can terminate even when the `bufio.ReadLine` is blocking, as we control the pipe. 
author_association: CONTRIBUTOR
user: login: ripienaar
created_at: 2025-04-30T09:34:00Z
body: OK, looking good and ^C behaves well, please rebase and we're ok here. 

Well schedule a revisit of the whole pub/sub code cos it's a bit of a mess - I mean before this PR already so no fault of this PR.  Will also land extensive unit testing for these commands then
author_association: COLLABORATOR
user: login: thewizzy
created_at: 2025-04-30T09:47:40Z
body: > OK, looking good and ^C behaves well, please rebase and we're ok here.
> 
> Well schedule a revisit of the whole pub/sub code cos it's a bit of a mess - I mean before this PR already so no fault of this PR. Will also land extensive unit testing for these commands then

I noticed whilst looking at testing that the timeout in `runNatsCli` does not work - I have a potential fix for nix. The other thing I noted is that there are a number of tests that do not work on Windows, partly because the `runNatsCli` command only supports windows (or bash on windows) and a few other tests that assume a nix environment. Many can be fixed by having a `runNatsCli` specifically for windows and wraping the command in `cmd /c` - though there are still a few tests that fail. Happy to dig more into it if that is welcomed. 

It should be rebased now. Thank you.
author_association: CONTRIBUTOR
user: login: ripienaar
created_at: 2025-04-30T09:56:53Z
body: Thank you @thewizzy we'll see what we can do about windows testing too /cc @ploubser 
author_association: COLLABORATOR