Repository: dankamongmen/notcurses (language: python)

Issue:
return unknown input escapes as discrete characters, not an error
If we read an escape sequence we can't match against our trie, we currently return an error. Instead, we need to play back the inputs as discrete characters. Users are likely to abort on an input error, and reasonably so.
return unknown input escapes as discrete characters, not an error
If we read an escape sequence we can't match against our trie, we currently return an error. Instead, we need to play back the inputs as discrete characters. Users are likely to abort on an input error, and reasonably so.
new input layer not returning bare NCKEY_ESC
The new input layer isn't returning a bare Escape as `NCKEY_ESC`, as it instead sits in the escape sequence processing automaton. Before, when we had an escape and no further input, we returned it. We probably want to keep this behavior. Reported by @kevinconley19.

This hasn't been released, but it's present in master.
new input layer not returning bare NCKEY_ESC
The new input layer isn't returning a bare Escape as `NCKEY_ESC`, as it instead sits in the escape sequence processing automaton. Before, when we had an escape and no further input, we returned it. We probably want to keep this behavior. Reported by @kevinconley19.

This hasn't been released, but it's present in master.


Discussion / hints:
so yeah in `process_melange()` we'll want to try and invoke `process_escape()` if we start our lexeme with an escape character. if it returns -1, feed that data into `process_bulk()`. if it returns 0, *and* we're at the end of our *virtual* input buffer, we need deal with a split escape. otherwise, treat a 0 like a -1 (we didn't have any more data available, or we would have read it, since we're not at the end). if it returns more than 0, consume them and iterate. at the end, move remaining characters to the front.

for the split escape case, we must immediately try a single non-blocking read. if that hits, restart the process. otherwise, feed the leftover data to the bulk processor.

yeah, that ought work. tricky, delicate. but it ought work.
so yeah in `process_melange()` we'll want to try and invoke `process_escape()` if we start our lexeme with an escape character. if it returns -1, feed that data into `process_bulk()`. if it returns 0, *and* we're at the end of our *virtual* input buffer, we need deal with a split escape. otherwise, treat a 0 like a -1 (we didn't have any more data available, or we would have read it, since we're not at the end). if it returns more than 0, consume them and iterate. at the end, move remaining characters to the front.

for the split escape case, we must immediately try a single non-blocking read. if that hits, restart the process. otherwise, feed the leftover data to the bulk processor.

yeah, that ought work. tricky, delicate. but it ought work.
as expected, it is returned when we redirect from a file. in that case, we're not looking for control sequences. the problem only appears when stdin is the terminal. so i think the thing to do is effectively exactly what we were doing before: consider a control sequence terminated when there is no further input available (this will require reading again when a possible sequence starts at the end of the buffer), combined with a solution to #2100, which we want fixed anyway.

so that means we need to keep a potential sequence around -- we can't entirely drop it and just rely on our position in the trie.
as expected, it is returned when we redirect from a file. in that case, we're not looking for control sequences. the problem only appears when stdin is the terminal. so i think the thing to do is effectively exactly what we were doing before: consider a control sequence terminated when there is no further input available (this will require reading again when a possible sequence starts at the end of the buffer), combined with a solution to #2100, which we want fixed anyway.

so that means we need to keep a potential sequence around -- we can't entirely drop it and just rely on our position in the trie.

Fix patch:
diff --git a/src/input/input.cpp b/src/input/input.cpp
index cf9ee886e1..41d3b994d6 100644
--- a/src/input/input.cpp
+++ b/src/input/input.cpp
@@ -309,6 +309,11 @@ int input_demo(ncpp::NotCurses* nc) {
         n->printf("Unicode: [0x%08x] '%lc'", r, (wchar_t)r);
       }
     }
+    int x;
+    n->get_cursor_yx(nullptr, &x);
+    for(int i = x ; i < n->get_dim_x() ; ++i){
+      n->putc(' ');
+    }
     if(!dim_rows(n)){
       break;
     }
diff --git a/src/lib/direct.c b/src/lib/direct.c
index 57701b0e54..16f24dccef 100644
--- a/src/lib/direct.c
+++ b/src/lib/direct.c
@@ -632,11 +632,7 @@ ncdirect_dump_plane(ncdirect* n, const ncplane* np, int xoff){
 int ncdirect_raster_frame(ncdirect* n, ncdirectv* ncdv, ncalign_e align){
   int lenx = ncplane_dim_x(ncdv);
   int xoff = ncdirect_align(n, align, lenx);
-  if(ncdirect_dump_plane(n, ncdv, xoff)){
-    free_plane(ncdv);
-    return -1;
-  }
-  int r = ncdirect_flush(n);
+  int r = ncdirect_dump_plane(n, ncdv, xoff);
   free_plane(ncdv);
   return r;
 }
@@ -1419,12 +1415,15 @@ int ncdirect_stream(ncdirect* n, const char* filename, ncstreamcb streamer,
       if(n->tcache.pixel_remove){
         fbuf f = {};
         fbuf_init_small(&f);
-        if(n->tcache.pixel_remove(lastid, &f) || fwrite(f.buf, f.used, 1, n->ttyfp) != 1){
+        if(n->tcache.pixel_remove(lastid, &f)){
           fbuf_free(&f);
           ncvisual_destroy(ncv);
           return -1;
         }
-        fbuf_free(&f);
+        if(fbuf_finalize(&f, n->ttyfp) < 0){
+          ncvisual_destroy(ncv);
+          return -1;
+        }
       }
     }
     streamer(ncv, vopts, NULL, curry);
diff --git a/src/lib/in.c b/src/lib/in.c
index 199cb49e76..299bbab8ff 100644
--- a/src/lib/in.c
+++ b/src/lib/in.c
@@ -163,6 +163,10 @@ typedef struct inputctx {
   tinfo* ti;          // link back to tinfo
   pthread_t tid;      // tid for input thread
 
+  unsigned midescape; // we're in the middle of a potential escape. we need
+                      //  to do a nonblocking read and try to complete it.
+  unsigned stdineof;  // have we seen an EOF on stdin?
+
   unsigned linesigs;  // are line discipline signals active?
   unsigned drain;     // drain away bulk input?
   ncsharedstats *stats; // stats shared with notcurses context
@@ -443,11 +447,13 @@ create_inputctx(tinfo* ti, FILE* infp, int lmargin, int tmargin,
                         i->initdata_complete = NULL;
                         i->stats = stats;
                         i->ti = ti;
+                        i->stdineof = 0;
                         i->ibufvalid = 0;
                         // FIXME need to get this out of the initial termios
                         // (as stored in tpreserved)
                         i->linesigs = 1;
                         i->tbufvalid = 0;
+                        i->midescape = 0;
                         i->numeric = 0;
                         i->stridx = 0;
                         i->runstring[i->stridx] = '\0';
@@ -596,7 +602,8 @@ prep_all_keys(inputctx* ictx){
 
 // populate |buf| with any new data from the specified file descriptor |fd|.
 static void
-read_input_nblock(int fd, unsigned char* buf, size_t buflen, int *bufused){
+read_input_nblock(int fd, unsigned char* buf, size_t buflen, int *bufused,
+                  unsigned* goteof){
   if(fd < 0){
     return;
   }
@@ -608,6 +615,11 @@ read_input_nblock(int fd, unsigned char* buf, size_t buflen, int *bufused){
   if(r <= 0){
     if(r < 0){
       logwarn("couldn't read from %d (%s)\n", fd, strerror(errno));
+    }else{
+      logwarn("got EOF on %d\n", fd);
+      if(goteof){
+        *goteof = 1;
+      }
     }
     return;
   }
@@ -803,9 +815,11 @@ set_sda_version(inputctx* ictx){
   return buf;
 }
 
-// ictx->numeric, ictx->p3, and ictx->p2 have the two parameters
+// ictx->numeric, ictx->p3, and ictx->p2 have the two parameters. we're using
+// SGR (1006) mouse encoding, so use the final character to determine release
+// ('M' for click, 'm' for release).
 static void
-mouse_click(inputctx* ictx){
+mouse_click(inputctx* ictx, unsigned release){
   // convert from 1- to 0-indexing, and account for margins
   const int x = ictx->p3 - 1 - ictx->lmargin;
   const int y = ictx->numeric - 1 - ictx->tmargin;
@@ -820,12 +834,10 @@ mouse_click(inputctx* ictx){
     return;
   }
   ncinput* ni = ictx->inputs + ictx->iwrite;
-  if(ictx->p2 >= 0 && ictx->p2 < 64){
-    if(ictx->p2 % 4 == 3){
-      ni->id = NCKEY_RELEASE;
-    }else{
-      ni->id = NCKEY_BUTTON1 + (ictx->p2 % 4);
-    }
+  if(release){
+    ni->id = NCKEY_RELEASE;
+  }else if(ictx->p2 >= 0 && ictx->p2 < 64){
+    ni->id = NCKEY_BUTTON1 + (ictx->p2 % 4);
   }else if(ictx->p2 >= 64 && ictx->p2 < 128){
     ni->id = NCKEY_BUTTON4 + (ictx->p2 % 4);
   }else if(ictx->p2 >= 128 && ictx->p2 < 192){
@@ -936,65 +948,70 @@ special_key(inputctx* ictx){
   pthread_cond_broadcast(&ictx->icond);
 }
 
-// ictx->numeric and ictx->p2 have the two parameters
+static void
+send_synth_signal(int sig){
+#ifndef __MINGW64__
+  if(sig){
+    raise(sig);
+  }
+#else
+  (void)synth; // FIXME
+#endif
+}
+
+// ictx->numeric and ictx->p2 have the two parameters, where ictx->numeric was
+// optional and indicates a special key with no modifiers.
 static void
 kitty_kbd(inputctx* ictx){
-  enum { // synthesized events derived from keypresses
-    SYNTH_NOTHING,
-    SYNTH_SIGINT,
-    SYNTH_SIGQUIT,
-  } synth = SYNTH_NOTHING;
-  assert(ictx->numeric > 0);
+  int synth = 0;
+  assert(ictx->numeric >= 0);
   assert(ictx->p2 > 0);
+  ncinput tni = {
+    .id = ictx->p2 == 0x7f ? NCKEY_BACKSPACE : ictx->p2,
+    .shift = !!((ictx->numeric - 1) & 0x1),
+    .alt = !!((ictx->numeric - 1) & 0x2),
+    .ctrl = !!((ictx->numeric - 1) & 0x4),
+  };
+  // FIXME decode remaining modifiers through 128
+  // standard keyboard protocol reports ctrl+ascii as the capital form,
+  // so (for now) conform when using kitty protocol...
+  if(tni.ctrl){
+    if(tni.id < 128 && islower(tni.id)){
+      tni.id = toupper(tni.id);
+    }
+    if(!tni.alt && !tni.shift){
+      if(tni.id == 'C'){
+        synth = SIGINT;
+      }else if(tni.id == '\\'){
+        synth = SIGQUIT;
+      }
+    }
+  }
+  tni.x = 0;
+  tni.y = 0;
   pthread_mutex_lock(&ictx->ilock);
   if(ictx->ivalid == ictx->isize){
     pthread_mutex_unlock(&ictx->ilock);
     logerror("dropping input 0x%08x 0x%02x\n", ictx->p2, ictx->numeric);
+    send_synth_signal(synth);
     return;
   }
   ncinput* ni = ictx->inputs + ictx->iwrite;
-  if((ni->id = ictx->p2) == 0x7f){
-    ni->id = NCKEY_BACKSPACE;
-  }
-  ni->shift = !!((ictx->numeric - 1) & 0x1);
-  ni->alt = !!((ictx->numeric - 1) & 0x2);
-  ni->ctrl = !!((ictx->numeric - 1) & 0x4);
-  // FIXME decode remaining modifiers through 128
-  // standard keyboard protocol reports ctrl+ascii as the capital form,
-  // so (for now) conform when using kitty protocol...
-  if(ni->id < 128 && islower(ni->id) && ni->ctrl){
-    ni->id = toupper(ni->id);
-  }
-  ni->x = 0;
-  ni->y = 0;
+  memcpy(ni, &tni, sizeof(tni));
   if(++ictx->iwrite == ictx->isize){
     ictx->iwrite = 0;
   }
   ++ictx->ivalid;
-  if(ni->ctrl && !ni->alt && !ni->shift){
-    if(ni->id == 'C'){
-      synth = SYNTH_SIGINT;
-    }else if(ni->id == '\\'){
-      synth = SYNTH_SIGQUIT;
-    }
-  }
   pthread_mutex_unlock(&ictx->ilock);
   pthread_cond_broadcast(&ictx->icond);
-#ifndef __MINGW64__
-  if(synth == SYNTH_SIGINT){
-    raise(SIGINT);
-  }else if(synth == SYNTH_SIGQUIT){
-    raise(SIGQUIT);
-  }
-#else
-  (void)synth; // FIXME
-#endif
+  send_synth_signal(synth);
 }
 
 // FIXME ought implement the full Williams automaton
 // FIXME sloppy af in general
 // returns 1 after handling the Device Attributes response, 0 if more input
 // ought be fed to the machine, and -1 on an invalid state transition.
+// returns 2 on a valid accept state that is not the final state.
 static int
 pump_control_read(inputctx* ictx, unsigned char c){
   logdebug("state: %2d char: %1c %3d %02x\n", ictx->state, isprint(c) ? c : ' ', c, c);
@@ -1039,6 +1056,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
     case STATE_APC_ST:
       if(c == '\\'){
         ictx->state = STATE_NULL;
+        return 2;
       }else{
         ictx->state = STATE_APC_DRAIN;
       }
@@ -1065,7 +1083,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
           return -1;
         }
         ictx->state = STATE_NULL;
-        break;
+        return 2;
       }
       ictx->numeric = c;
       if(ruts_string(ictx, STATE_BG1)){
@@ -1097,6 +1115,13 @@ pump_control_read(inputctx* ictx, unsigned char c){
         ictx->p2 = ictx->numeric;
         ictx->state = STATE_CURSOR_COL;
         ictx->numeric = 0;
+      }else if(c == 'u'){
+        // kitty keyboard protocol
+        ictx->p2 = ictx->numeric;
+        ictx->numeric = 0;
+        kitty_kbd(ictx);
+        ictx->state = STATE_NULL;
+        return 2;
       }else if(c >= 0x40 && c <= 0x7E){
         ictx->state = STATE_NULL;
       }
@@ -1133,8 +1158,13 @@ pump_control_read(inputctx* ictx, unsigned char c){
           return -1;
         }
       }else if(c == 'M'){
-        mouse_click(ictx);
+        mouse_click(ictx, 0);
+        ictx->state = STATE_NULL;
+        return 2;
+      }else if(c == 'm'){
+        mouse_click(ictx, 1);
         ictx->state = STATE_NULL;
+        return 2;
       }else{
         ictx->state = STATE_NULL;
       }
@@ -1168,6 +1198,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
           }
         }
         ictx->state = STATE_NULL;
+        return 2;
       }else if(c == 't'){
 //fprintf(stderr, "CELLS X: %d\n", ictx->numeric);
         if(ictx->initdata){
@@ -1175,10 +1206,12 @@ pump_control_read(inputctx* ictx, unsigned char c){
           ictx->initdata->dimy = ictx->p2;
         }
         ictx->state = STATE_NULL;
+        return 2;
       }else if(c == 'u'){
         // kitty keyboard protocol
         kitty_kbd(ictx);
         ictx->state = STATE_NULL;
+        return 2;
       }else if(c == ';'){
         if(ictx->p2 == 4){
           if(ictx->initdata){
@@ -1211,6 +1244,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
           loginfo("got pixel geometry: %d/%d\n", ictx->initdata->pixy, ictx->initdata->pixx);
         }
         ictx->state = STATE_NULL;
+        return 2;
       }else{
         ictx->state = STATE_NULL;
       }
@@ -1226,6 +1260,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
           loginfo("got cell geometry: %d/%d\n", ictx->initdata->dimy, ictx->initdata->dimx);
         }
         ictx->state = STATE_NULL;
+        return 2;
       }else{
         ictx->state = STATE_NULL;
       }
@@ -1359,6 +1394,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
     case STATE_SDA_DRAIN:
       if(c == 'c'){
         ictx->state = STATE_NULL;
+        return 2;
       }
       break;
     // primary device attributes and XTSMGRAPHICS replies are generally
@@ -1438,6 +1474,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
           ictx->numeric = 0;
         }
         ictx->state = STATE_NULL;
+        return 2;
       }else if(c >= 0x40 && c <= 0x7E){
         ictx->state = STATE_NULL;
         if(c == 'c'){
@@ -1466,6 +1503,8 @@ pump_control_read(inputctx* ictx, unsigned char c){
           ictx->initdata->sixelx = ictx->p4;
           ictx->initdata->sixely = ictx->numeric;
           loginfo("max sixel geometry: %dx%d\n", ictx->initdata->sixely, ictx->initdata->sixelx);
+          ictx->state = STATE_NULL;
+          return 2;
         }
       }else if(c >= 0x40 && c <= 0x7E){
         ictx->state = STATE_NULL;
@@ -1485,6 +1524,7 @@ pump_control_read(inputctx* ictx, unsigned char c){
     case STATE_APPSYNC_REPORT_DRAIN:
       if(c == 'y'){
         ictx->state = STATE_NULL;
+        return 2;
       }
       break;
     default:
@@ -1503,56 +1543,114 @@ handoff_initial_responses(inputctx* ictx){
   pthread_cond_broadcast(&ictx->icond);
 }
 
-// try to lex a control sequence off of buf. return the number of bytes
+// try to lex a single control sequence off of buf. return the number of bytes
 // consumed if we do so, and -1 otherwise. buf is *not* necessarily
-// NUL-terminated. precondition: buflen >= 1.
-// FIXME we ought play complete failures into the general input buffer?
+// NUL-terminated. if we are definitely *not* an escape, or we're unsure when
+// we run out of input, return the negated relevant number of bytes, setting
+// ictx->midescape if we're uncertain.
+//  precondition: buflen >= 1. precondition: buf[0] == 0x1b.
 static int
 process_escape(inputctx* ictx, const unsigned char* buf, int buflen){
+  assert(1 <= buflen);
+  // FIXME given that we keep our state across invocations, we want buf offset
+  //  by the amount we handled the last iteration, if any was left over...
+  //  until then, we reset the state on entry. remove that once we preserve!
+  if(buf[0] != '\x1b'){
+    return -1;
+  }
+  ictx->triepos = ictx->inputescapes;
+  ictx->state = STATE_NULL;
+  // FIXME end material eliminated by maintaining state + offset
   int used = 0;
+  // FIXME we really want to unite these two automata, ugh
   while(used < buflen){
-    int r = pump_control_read(ictx, buf[used]);
-    if(r == 1){
+    unsigned char candidate = buf[used++];
+    int r = pump_control_read(ictx, candidate);
+    if(r == 2){
+      return used;
+    }
+    if(r == 1){ // received DA1, ending initial responses
+      // safe to call even if initdata == NULL
       handoff_initial_responses(ictx);
+      return used;
     }
-    // FIXME this drops alt+ characters intermingled with responses
-    if(ictx->initdata == NULL){
-      // an escape always resets the trie, as does a NULL transition
-      unsigned char candidate = buf[used];
-      if(candidate == NCKEY_ESC){
-        ictx->triepos = ictx->inputescapes;
-      }else if(ictx->triepos->trie[candidate] == NULL){
-        if(ictx->state == STATE_ESC){
-          if(candidate && candidate <= 0x80){ // FIXME what about supraASCII utf8?
-            alt_key(ictx, candidate);
-          }
+    if(ictx->initdata){
+      continue; // FIXME drops any Alt+chars entwined within initial responses
+    }
+    // an escape always resets the trie, as does a NULL transition
+    if(candidate == NCKEY_ESC){
+      ictx->triepos = ictx->inputescapes;
+      if(used > 1){ // we got reset; replay as input
+        return -(used - 1);
+      }
+      // validated first byte as escape! keep going. otherwise, check trie.
+      // we can safely check trie[candidate] above because we are either coming
+      // off the initial node, which definitely has a valid ->trie, or we're
+      // coming from a transition, where ictx->triepos->trie is checked below.
+    }else if(ictx->triepos->trie[candidate] == NULL){
+      ictx->triepos = ictx->inputescapes;
+      if(ictx->state == STATE_ESC){
+        if(candidate && candidate <= 0x80){ // FIXME what about supraASCII utf8?
+          alt_key(ictx, candidate);
+          return used;
         }
+      }
+      if(ictx->state == STATE_NULL){
+        // we were an invalid sequence; replay as input. we know used > 1. the
+        // current character is not yet excluded, so return negative "used - 1".
+        return -(used - 1);
+      }
+    }else{
+      ictx->triepos = ictx->triepos->trie[candidate];
+      logtrace("triepos: %p in: %u special: 0x%08x\n", ictx->triepos, candidate, ictx->triepos->special);
+      if(ictx->triepos->special != NCKEY_INVALID){ // match! mark and reset
+        special_key(ictx);
         ictx->triepos = ictx->inputescapes;
-      }else{
-        ictx->triepos = ictx->triepos->trie[candidate];
-        logtrace("triepos: %p in: %u special: 0x%08x\n", ictx->triepos, candidate, ictx->triepos->special);
-        if(ictx->triepos->special != NCKEY_INVALID){ // match! mark and reset
-          special_key(ictx);
-          ictx->triepos = ictx->inputescapes;
-        }else if(ictx->triepos->trie == NULL){
+        return used;
+      }else if(ictx->triepos->trie == NULL){
+        if(ictx->state == STATE_NULL){
           ictx->triepos = ictx->inputescapes;
+          // all inspected characters are invalid; return full negative "used"
+          return -used;
         }
       }
     }
-    ++used;
   }
-  return used;
+  // we exhausted input without knowing whether or not this is a valid control
+  // sequence; we're still on-trie, and need more (immediate) input.
+  ictx->midescape = 1;
+  return -used;
 }
 
 // process as many control sequences from |buf|, having |bufused| bytes,
 // as we can. anything not a valid control sequence is dropped. this text
-// needn't be valid UTF-8.
+// needn't be valid UTF-8. this is always called on tbuf; if we find bulk data
+// here, we need replay it into ibuf (assuming that there's room).
 static void
 process_escapes(inputctx* ictx, unsigned char* buf, int* bufused){
   int offset = 0;
   while(*bufused){
     int consumed = process_escape(ictx, buf + offset, *bufused);
+    // if we aren't certain, that's not a control sequence unless we're at
+    // the end of the tbuf, in which case we really do try reading more. if
+    // this was not a sequence, we'll catch it on the next read.
     if(consumed < 0){
+      int tavailable = sizeof(ictx->tbuf) - (offset + *bufused - consumed);
+      if(!ictx->midescape || tavailable){
+        consumed = -consumed;
+        int available = sizeof(ictx->ibuf) - ictx->ibufvalid;
+        if(available){
+          if(available > consumed){
+            available = consumed;
+          }
+          logwarn("replaying %dB of %dB to ibuf\n", available, consumed);
+          memcpy(ictx->ibuf + ictx->ibufvalid, buf + offset, available);
+          ictx->ibufvalid += available;
+        }
+        *bufused -= consumed;
+        offset += consumed;
+        ictx->midescape = 0;
+      }
       break;
     }
     *bufused -= consumed;
@@ -1564,16 +1662,23 @@ process_escapes(inputctx* ictx, unsigned char* buf, int* bufused){
   }
 }
 
-// precondition: buflen >= 1.
+// precondition: buflen >= 1. attempts to consume UTF8 input from buf. the
+// expected length of a UTF8 character can be determined from its first byte.
+// if we don't have that much data, return 0 and read more. if we determine
+// an error, return -1 to consume 1 byte, restarting the UTF8 lex on the next
+// byte. on a valid UTF8 character, set up the ncinput and return its length.
 static int
 process_input(const unsigned char* buf, int buflen, ncinput* ni){
+  assert(1 <= buflen);
   memset(ni, 0, sizeof(*ni));
-  if(buf[0] == 0x7f){ // ASCII del, treated as backspace
-    ni->id = NCKEY_BACKSPACE;
-    return 1;
-  }
-  if(buf[0] < 0x80){ // pure ascii can't show up mid-utf8
-    if(buf[0] == '\n' || buf[0] == '\r'){
+  const int cpointlen = utf8_codepoint_length(*buf);
+  if(cpointlen <= 0){
+    logwarn("invalid UTF8 initiator on input (0x%02x)\n", *buf);
+    return -1;
+  }else if(cpointlen == 1){ // pure ascii can't show up mid-utf8-character
+    if(buf[0] == 0x7f){ // ASCII del, treated as backspace
+      ni->id = NCKEY_BACKSPACE;
+    }else if(buf[0] == '\n' || buf[0] == '\r'){
       ni->id = NCKEY_ENTER;
     }else if(buf[0] > 0 && buf[0] <= 26 && buf[0] != '\t'){
       ni->id = buf[0] + 'A' - 1;
@@ -1583,23 +1688,21 @@ process_input(const unsigned char* buf, int buflen, ncinput* ni){
     }
     return 1;
   }
-  int cpointlen = 0;
+  if(cpointlen > buflen){
+    logwarn("utf8 character (%dB) broken across read\n", cpointlen);
+    return 0; // need read more data; we don't have the complete character
+  }
   wchar_t w;
-  mbstate_t mbstate;
-  // FIXME verify that we have enough length based off first char
-  while(++cpointlen <= (int)MB_CUR_MAX && cpointlen < buflen){
+  mbstate_t mbstate = {};
 //fprintf(stderr, "CANDIDATE: %d cpointlen: %zu cpoint: %d\n", candidate, cpointlen, cpoint[cpointlen]);
-    // FIXME how the hell does this work with 16-bit wchar_t?
-    memset(&mbstate, 0, sizeof(mbstate));
-    size_t r;
-    if((r = mbrtowc(&w, (const char*)buf, cpointlen + 1, &mbstate)) != (size_t)-1 &&
-        r != (size_t)-2){
-      ni->id = w;
-      return cpointlen + 1;
-    }
+  // FIXME how the hell does this work with 16-bit wchar_t?
+  size_t r = mbrtowc(&w, (const char*)buf, cpointlen, &mbstate);
+  if(r == (size_t)-1 || r == (size_t)-2){
+    logerror("invalid utf8 prefix (%dB) on input\n", cpointlen);
+    return -1;
   }
-  // FIXME input error stat
-  return 0;
+  ni->id = w;
+  return cpointlen;
 }
 
 // precondition: buflen >= 1. gets an ncinput prepared by process_input, and
@@ -1622,6 +1725,9 @@ process_ncinput(inputctx* ictx, const unsigned char* buf, int buflen){
       }
       ++ictx->ivalid;
     }
+  }else if(r < 0){
+    inc_input_errors(ictx);
+    r = 1; // we want to consume a single byte, upstairs
   }
   pthread_mutex_unlock(&ictx->ilock);
   pthread_cond_broadcast(&ictx->icond);
@@ -1661,7 +1767,20 @@ process_melange(inputctx* ictx, const unsigned char* buf, int* bufused){
     int consumed = 0;
     if(buf[offset] == '\x1b'){
       consumed = process_escape(ictx, buf + offset, *bufused);
-    }else{
+      if(consumed < 0){
+        if(ictx->midescape){
+          if(offset + *bufused - consumed != sizeof(ictx->ibuf)){
+            // not at the end; treat it as input. no need to move between
+            // buffers; simply ensure we process it as input, and don't mark
+            // anything as consumed.
+            ictx->midescape = 0;
+          }
+        }
+      }
+    }
+    // don't process as input only if we just read a valid control character,
+    // or if we need to read more to determine what it is.
+    if(consumed <= 0 && !ictx->midescape){
       consumed = process_ncinput(ictx, buf + offset, *bufused);
     }
     if(consumed < 0){
@@ -1706,13 +1825,22 @@ int ncinput_shovel(inputctx* ictx, const void* buf, int len){
   return 0;
 }
 
+// here, we always block for an arbitrarily long time, or not at all,
+// doing the latter only when ictx->midescape is set. |rtfd| and/or |rifd|
+// are set high iff they are ready for reading, and otherwise cleared.
 static int
-block_on_input(inputctx* ictx){
-  struct timespec* ts = NULL; // FIXME
+block_on_input(inputctx* ictx, unsigned* rtfd, unsigned* rifd){
+  *rtfd = *rifd = 0;
+  unsigned nonblock = ictx->midescape;
+  if(nonblock){
+    loginfo("nonblocking read to check for completion\n");
+    ictx->midescape = 0;
+  }
 #ifdef __MINGW64__
-	int timeoutms = ts ? ts->tv_sec * 1000 + ts->tv_nsec / 1000000 : -1;
+	int timeoutms = nonblock ? 0 : -1;
 	DWORD d = WaitForMultipleObjects(1, &ictx->stdinhandle, FALSE, timeoutms);
 	if(d == WAIT_TIMEOUT){
+    *rifd = 1;
 		return 0;
 	}else if(d == WAIT_FAILED){
 		return -1;
@@ -1725,30 +1853,40 @@ block_on_input(inputctx* ictx){
 #ifdef POLLRDHUP
   inevents |= POLLRDHUP;
 #endif
-  struct pollfd pfds[2] = {
-    {
-			.fd = ictx->stdinfd,
-			.events = inevents,
-			.revents = 0,
+  struct pollfd pfds[2];
+  int pfdcount = 0;
+  if(!ictx->stdineof){
+    if(ictx->ibufvalid != sizeof(ictx->ibuf)){
+      pfds[pfdcount].fd = ictx->stdinfd;
+      pfds[pfdcount].events = inevents;
+      pfds[pfdcount].revents = 0;
+      ++pfdcount;
     }
-  };
-  int pfdcount = 1;
+  }
   if(ictx->termfd >= 0){
     pfds[pfdcount].fd = ictx->termfd;
     pfds[pfdcount].events = inevents;
     pfds[pfdcount].revents = 0;
     ++pfdcount;
   }
-  int events;
-#if defined(__APPLE__) || defined(__MINGW64__)
-  int timeoutms = ts ? ts->tv_sec * 1000 + ts->tv_nsec / 1000000 : -1;
-  while((events = poll(pfds, pfdcount, timeoutms)) < 0){ // FIXME smask?
-#else
   sigset_t smask;
   sigfillset(&smask);
   sigdelset(&smask, SIGCONT);
   sigdelset(&smask, SIGWINCH);
-  while((events = ppoll(pfds, pfdcount, ts, &smask)) < 0){
+  if(pfdcount == 0){
+    loginfo("output queues full; blocking on signals\n");
+    int signum;
+    sigwait(&smask, &signum);
+    return 0;
+  }
+  int events;
+#if defined(__APPLE__) || defined(__MINGW64__)
+	int timeoutms = nonblock ? 0 : -1;
+  while((events = poll(pfds, pfdcount, timeoutms)) < 0){ // FIXME smask?
+#else
+  struct timespec ts = { .tv_sec = 0, .tv_nsec = 0, };
+  struct timespec* pts = nonblock ? &ts : NULL;
+  while((events = ppoll(pfds, pfdcount, pts, &smask)) < 0){
 #endif
     if(errno != EAGAIN && errno != EBUSY && errno != EWOULDBLOCK){
       return -1;
@@ -1756,6 +1894,18 @@ block_on_input(inputctx* ictx){
       return resize_seen;
     }
   }
+  pfdcount = 0;
+  while(events){
+    if(pfds[pfdcount].revents){
+      if(pfds[pfdcount].fd == ictx->stdinfd){
+        *rifd = 1;
+      }else if(pfds[pfdcount].fd == ictx->termfd){
+        *rtfd = 1;
+      }
+      --events;
+    }
+  }
+loginfo("got events: %c%c %d\n", *rtfd ? 'T' : 't', *rifd ? 'I' : 'i', pfds[0].revents);
   return events;
 #endif
 }
@@ -1764,16 +1914,21 @@ block_on_input(inputctx* ictx){
 // don't loop around this call without some kind of readiness notification.
 static void
 read_inputs_nblock(inputctx* ictx){
+  unsigned rtfd, rifd;
   // FIXME also need to wake up if our output queues have space that has
   // opened up for us to write into, lest we deadlock with full buffers...
-  block_on_input(ictx);
+  block_on_input(ictx, &rtfd, &rifd);
   // first we read from the terminal, if that's a distinct source.
-  read_input_nblock(ictx->termfd, ictx->tbuf, sizeof(ictx->tbuf),
-                    &ictx->tbufvalid);
+  if(rtfd){
+    read_input_nblock(ictx->termfd, ictx->tbuf, sizeof(ictx->tbuf),
+                      &ictx->tbufvalid, NULL);
+  }
   // now read bulk, possibly with term escapes intermingled within (if there
   // was not a distinct terminal source).
-  read_input_nblock(ictx->stdinfd, ictx->ibuf, sizeof(ictx->ibuf),
-                    &ictx->ibufvalid);
+  if(rifd){
+    read_input_nblock(ictx->stdinfd, ictx->ibuf, sizeof(ictx->ibuf),
+                      &ictx->ibufvalid, &ictx->stdineof);
+  }
 }
 
 static void*
diff --git a/src/lib/termdesc.c b/src/lib/termdesc.c
index 3399b805b5..074e57c9c4 100644
--- a/src/lib/termdesc.c
+++ b/src/lib/termdesc.c
@@ -785,7 +785,8 @@ int interrogate_terminfo(tinfo* ti, const char* termtype, FILE* out, unsigned ut
   // machines, but they'll use the terminfo installed thereon (putty, etc.).
   int termerr;
   if(setupterm(termtype, ti->ttyfd, &termerr)){
-    logpanic("Terminfo error %d for %s (see terminfo(3ncurses))\n", termerr, termtype);
+    logpanic("Terminfo error %d for [%s] (see terminfo(3ncurses))\n",
+             termerr, termtype ? termtype : "");
     goto err;
   }
   tname = termname(); // longname() is also available
diff --git a/src/lib/termdesc.h b/src/lib/termdesc.h
index 33802e502d..a3c08399b9 100644
--- a/src/lib/termdesc.h
+++ b/src/lib/termdesc.h
@@ -212,7 +212,7 @@ int interrogate_terminfo(tinfo* ti, const char* termtype, FILE* out,
                          unsigned nonewfonts, int* cursor_y, int* cursor_x,
                          struct ncsharedstats* stats, int lmargin, int tmargin,
                          unsigned draininput)
-  __attribute__ ((nonnull (1, 2, 3, 10)));
+  __attribute__ ((nonnull (1, 3, 10)));
 
 void free_terminfo_cache(tinfo* ti);
 


Test patch:
diff --git a/src/tests/main.cpp b/src/tests/main.cpp
index 77c89af33d..d011fc5857 100644
--- a/src/tests/main.cpp
+++ b/src/tests/main.cpp
@@ -86,6 +86,7 @@ reset_terminal(){
       tios.c_lflag |= ISIG | ICANON | ECHO;
       tcsetattr(fd, TCSADRAIN, &tios);
     }
+    printf("\x1b[<u"); // pop any kitty keyboard state we set
     char* str = tigetstr("sgr0");
     if(str != (char*)-1){
       printf("%s", str);
diff --git a/tools/interactive-tester.sh b/tools/interactive-tester.sh
new file mode 100755
index 0000000000..ae9cfb90e3
--- /dev/null
+++ b/tools/interactive-tester.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+
+set -e
+
+# interactive tester for Notcurses. runs a variety of programs, which the
+# user must inspect for correct output -- i.e. these do not check their
+# own output for correctness =[. it ought be run from a notcurses build
+# directory, with all binaries built.
+
+DATA=../data # FIXME
+OUT=$(basename 0).log # FIXME
+rm -f "$OUT"
+
+[ -d "$DATA" ] || { echo "$DATA was not a directory" >&2 ; exit 1 ; }
+
+# this ought process the entire file, then allow the user to exit with ^D
+./notcurses-input -v < ../src/lib/in.c 2>>"$OUT"
+
+# goes through a series of self-described bitmap frames
+./bitmapstates 2>>"$OUT"
+
+./statepixel "$DATA"/worldmap.jpg 2>>"$OUT"
+
+./ncneofetch -v 2>>"$OUT"