Fix trace file fails on powerpc64

I see the following fails on powerpc64-linux,

(gdb) target tfile tfile-basic.tf^M
warning: Uploaded tracepoint 1 has no source location, using raw address^M
Tracepoint 1 at 0x10012358^M
Created tracepoint 1 for target's tracepoint 1 at 0x10012358.^M
(gdb) PASS: gdb.trace/tfile.exp: target tfile tfile-basic.tf
info trace^M
Num     Type           Disp Enb Address            What^M
1       tracepoint     keep y   0x0000000010012358 <write_basic_trace_file>^M
        installed on target^M
(gdb) FAIL: gdb.trace/tfile.exp: info tracepoints on trace file

-target-select tfile tfile-basic.tf^M
=thread-group-started,id="i1",pid="1"^M
=thread-created,id="1",group-id="i1"^M
&"warning: Uploaded tracepoint 1 has no source location, using raw address\n"^M
=breakpoint-created,bkpt={number="1",type="tracepoint",disp="keep",enabled="y",
addr="0x0000000010012358",at="<write_basic_trace_file>",thread-groups=["i1"],
times="0",installed="y",original-location="*0x10012358"}^M
~"Created tracepoint 1 for target's tracepoint 1 at 0x10012358.\n"^M
^connected^M
(gdb) ^M
FAIL: gdb.trace/mi-traceframe-changed.exp: tfile: select trace file

These fails are caused by writing function descriptor address into trace
file instead of function address.  This patch is to teach tfile.c to
write function address on powerpc64 target.  With this patch applied,
fails in tfile.exp and mi-traceframe-changed.exp are fixed.  Is it
OK?

gdb/testsuite:

2014-10-27  Yao Qi  <yao@codesourcery.com>

	* gdb.trace/tfile.c (adjust_function_address)
	[__powerpc64__ && _CALL_ELF != 2]: Get function address from
	function descriptor.
This patch closes a potential security hole in applications that use
the bfd library to parse binaries containing maliciously corrupt section
group headers.

	PR binutils/17510
	* elf.c (setup_group): Improve handling of corrupt group
	sections.
Fix a seg-fault in strings and other binutuils when parsing a corrupt PE
executable with an invalid value in the NumberOfRvaAndSizes field of the
AOUT header.

	PR binutils/17512
	* peXXigen.c (_bfd_XXi_swap_aouthdr_in): Handle corrupt binaries
	with an invalid value for NumberOfRvaAndSizes.
This fixes more seg-faults in tools like "strings" and "objdump" when
presented with corrupt binaries.

	PR binutils/17512
	* elf.c (bfd_section_from_shdr): Detect and warn about ELF
	binaries with a group of sections linked by the string table
	indicies.
	* peXXigen.c (pe_print_edata): Detect out of range rvas and
	entry counts for the Export Address table, Name Pointer table
	 and Ordinal table.
stepi/nexti: skip signal handler if "handle nostop" signal arrives

I noticed that "si" behaves differently when a "handle nostop" signal
arrives while the step is in progress, depending on whether the
program was stopped at a breakpoint when "si" was entered.
Specifically, in case GDB needs to step off a breakpoint, the handler
is skipped and the program stops in the next "mainline" instruction.
Otherwise, the "si" stops in the first instruction of the signal
handler.

I was surprised the testsuite doesn't catch this difference.  Turns
out gdb.base/sigstep.exp covers a bunch of cases related to stepping
and signal handlers, but does not test stepi nor nexti, only
step/next/continue.

My first reaction was that stopping in the signal handler was the
correct thing to do, as it's where the next user-visible instruction
that is executed is.  I considered then "nexti" -- a signal handler
could be reasonably considered a subroutine call to step over, it'd
seem intuitive to me that "nexti" would skip it.

But then, I realized that signals that arrive while a plain/line
"step" is in progress _also_ have their handler skipped.  A user might
well be excused for being confused by this, given:

  (gdb) help step
  Step program until it reaches a different source line.

And the signal handler's sources will be in different source lines,
after all.

I think that having to explain that "stepi" steps into handlers, (and
that "nexti" wouldn't according to my reasoning above), while "step"
does not, is a sign of an awkward interface.

E.g., if a user truly is interested in stepping into signal handlers,
then it's odd that she has to either force the signal to "handle
stop", or recall to do "stepi" whenever such a signal might be
delivered.  For that use case, it'd seem nicer to me if "step" also
stepped into handlers.

This suggests to me that we either need a global "step-into-handlers"
setting, or perhaps better, make "handle pass/nopass stop/nostop
print/noprint" have have an additional axis - "handle
stepinto/nostepinto", so that the user could configure whether
handlers for specific signals should be stepped into.

In any case, I think it's simpler (and thus better) for all step
commands to behave the same.  This commit thus makes "si/ni" skip
handlers for "handle nostop" signals that arrive while the command was
already in progress, like step/next do.

To be clear, nothing changes if the program was stopped for a signal,
and the user enters a stepping command _then_ -- GDB still steps into
the handler.  The change concerns signals that don't cause a stop and
that arrive while the step is in progress.

Tested on x86_64 Fedora 20, native and gdbserver.

gdb/
2014-10-27  Pedro Alves  <palves@redhat.com>

	* infrun.c (handle_signal_stop): Also skip handlers when a random
	signal arrives while handling a "stepi" or a "nexti".  Set the
	thread's 'step_after_step_resume_breakpoint' flag.

gdb/doc/
2014-10-27  Pedro Alves  <palves@redhat.com>

	* gdb.texinfo (Continuing and Stepping): Add cross reference to
	info on stepping and signal handlers.
	(Signals): Explain stepping and signal handlers.  Add context
	index entry, and cross references.

gdb/testsuite/
2014-10-27  Pedro Alves  <palves@redhat.com>

	* gdb.base/sigstep.c (dummy): New global.
	(main): Issue a couple writes to the new global.
	* gdb.base/sigstep.exp (get_next_pc, test_skip_handler): New
	procedures.
	(skip_over_handler): Use test_skip_handler.
	(top level): Call skip_over_handler for stepi and nexti too.
	(breakpoint_over_handler): Use test_skip_handler.
	(top level): Call breakpoint_over_handler for stepi and nexti too.
daily update
This patch fixes a flaw in the SREC parser which could cause a stack overflow
and potential secuiryt breach.

	PR binutils/17510
	* srec.c (srec_bad_byte): Increase size of buf to allow for
	negative values.
	(srec_scan): Use an unsigned char buffer to hold header bytes.
Workaround remote targets that report an empty list to qfThreadInfo

In https://sourceware.org/ml/gdb-patches/2014-10/msg00652.html, Sandra
shows a target that was broken by the recent update_thread_list
optimization:

 (gdb) target remote qa8-centos32-cs:10514
 ...
 (gdb) continue
 Continuing.
 Cannot execute this command without a live selected thread.
 (gdb)

The error means that the current thread is in "exited" state when the
continue command is processed.  The root of the problem was found
here:

 > Sending packet: $Hg0#df...Packet received:
 ...
 > Sending packet: $?#3f...Packet received: S00
 > Sending packet: $qfThreadInfo#bb...Packet received: l
 > Sending packet: $Hc-1#09...Packet received:
 > Sending packet: $qC#b4...Packet received: unset

This target doesn't really support threads (no thread indication in
stop reply packets; no support for qC), but then supports
qfThreadInfo, and returns an empty thread list to GDB.

See https://sourceware.org/ml/gdb-patches/2014-10/msg00665.html for
why the target does that.

As remote_update_thread_list deletes threads from GDB's list that are
not found in the thread list that the target reports, the result is
that GDB deletes the "fake" main thread that GDB added itself.  (As
that thread is currently selected, it is marked "exited" instead of
being deleted straight away.)

This commit avoids deleting the main thread in this scenario.

gdb/
2014-10-27  Pedro Alves  <palves@redhat.com>

	* remote.c (remote_thread_alive): New, factored out from ...
	(remote_thread_alive): ... this.
	(remote_update_thread_list): Bail out before deleting threads if
	the target returned an empty list, and, the current thread has a
	magic/fake ptid.
update bug numbers (GNATS -> Bugzilla) in a few signal related tests

This makes it easier to find the bugs in Bugzilla.

gdb/testsuite/
2014-10-28  Pedro Alves  <palves@redhat.com>

	* gdb.base/sigaltstack.exp: Update to use Bugzilla bug numbers
	instead of GNATS numbers.
	* gdb.base/sigbpt.exp: Likewise.
	* gdb.base/siginfo.exp: Likewise.
	* gdb.base/sigstep.exp: Likewise.
gdb.base/sigstep.exp: cleanup and make it easier to extend

Hacking on sigstep.exp, I found it harder to understand and extend
than ideal.

 - GDB is currently not restarted between the different
   tests/combinations in the file, and some parts of the tests' setup
   are done on the top level, and shared between tests.  It's not
   trivial to understand which breakpoints each test procedure expects
   to be set or not set.  And it's not trivial to disable parts of the
   test if you want quickly try out just a subset of the tests
   (running the whole file takes a bit).

 - Because GDB is currently not restarted between tests, if some test
   triggers a ptrace/kernel bug, the following tests may end up with
   cascading fails.  That makes it hard to add a test to cover a
   kernel bug that isn't fixed yet, with a xfail/kfail.  E.g,. note
   how with kernels with bug gdb/8744 (stepi over sigreturn syscall
   exits program) the test program exits, and nothing restarts it
   afterwards...

 - The manual test message prefix management gets a bit in the way.
   Nowadays, we have with_test_prefix which makes it simpler.

 - 'i' is used as parameter name in the various procedures, meaning
   'the command the test', which isn't as obvious as it could.

This commit addresses all that.

gdb/testsuite/
2014-10-28  Pedro Alves  <palves@redhat.com>

	* gdb.base/sigstep.exp: Use build_executable instead of
	prepare_for_testing.
	(top level): Move code that starts GDB, runs to main and creates a
	display to ...
	(restart): ... this new procedure.
	(top level): Move backtrace from signal handler test to ...
	(validate_backtrace): ... this new procedure.
	(advance, advancei): Rename parameter from 'i' to 'cmd'.  Use
	with_test_prefix.  Always restart GDB.
	(skip_to_handler): Rename parameter from 'i' to 'cmd'.  Use
	with_test_prefix.  Always restart GDB.  No need to delete
	breakpoints after the test.
	(test_skip_handler): Remove prefix parameter.
	(skip_over_handler, breakpoint_to_handler)
	(breakpoint_to_handler_entry, breakpoint_over_handler): Rename
	parameter from 'i' to 'cmd'.  Use with_test_prefix.  Always
	restart GDB.  No need to delete breakpoints after the test.
	(top level): Use foreach to call the test procedures with
	different commands.
More fixes for corrupt binaries crashing the binutils.

	PR binutils/17512
	* elf.c (bfd_section_from_shdr): Allocate and free the recursion
	detection table on a per-bfd basis.
	* peXXigen.c (pe_print_edata): Handle binaries with a truncated
	export table.
Test for PR gdb/17511, spurious SIGTRAP after stepping into+in signal handler

I noticed that when I single-step into a signal handler with a
pending/queued signal, the following single-steps while the program is
in the signal handler leave $eflags.TF set.  That means subsequent
continues will trap after one instruction, resulting in a spurious
SIGTRAP being reported to the user.

This is a kernel bug; I've reported it to kernel devs (turned out to
be a known bug).  I'm seeing it on x86_64 Fedora 20 (Linux
3.16.4-200.fc20.x86_64), and I was told it's still not fixed upstream.

This commit extends gdb.base/sigstep.exp to cover this use case,
xfailed.

Here's what the bug looks like:

 (gdb) start
 Temporary breakpoint 1, main () at si-handler.c:48
 48        setup ();
 (gdb) next
 50        global = 0; /* set break here */

Let's queue a signal, so we can step into the handler:

 (gdb) handle SIGUSR1
 Signal        Stop      Print   Pass to program Description
 SIGUSR1       Yes       Yes     Yes             User defined signal 1
 (gdb) queue-signal SIGUSR1

TF is not set:

 (gdb) display $eflags
 1: $eflags = [ PF ZF IF ]

Now step into the handler -- "si" does PTRACE_SINGLESTEP+SIGUSR1:

 (gdb) si
 sigusr1_handler (sig=0) at si-handler.c:31
 31      {
 1: $eflags = [ PF ZF IF ]

No TF yet.  But another single-step...

 (gdb) si
 0x0000000000400621      31      {
 1: $eflags = [ PF ZF TF IF ]

... ends up with TF left set.  This results in PTRACE_CONTINUE
trapping after each instruction is executed:

 (gdb) c
 Continuing.

 Program received signal SIGTRAP, Trace/breakpoint trap.
 0x0000000000400624 in sigusr1_handler (sig=0) at si-handler.c:31
 31      {
 1: $eflags = [ PF ZF TF IF ]

 (gdb) c
 Continuing.

 Program received signal SIGTRAP, Trace/breakpoint trap.
 sigusr1_handler (sig=10) at si-handler.c:32
 32        global = 0;
 1: $eflags = [ PF ZF TF IF ]
 (gdb)

Note that even another PTRACE_SINGLESTEP does not fix it:

 (gdb) si
 33      }
 1: $eflags = [ PF ZF TF IF ]
 (gdb)

Eventually, it gets "fixed" by the rt_sigreturn syscall, when
returning out of the handler:

 (gdb) bt
 #0  sigusr1_handler (sig=10) at si-handler.c:33
 #1  <signal handler called>
 #2  main () at si-handler.c:50
 (gdb) set disassemble-next-line on
 (gdb) si
 0x0000000000400632      33      }
    0x0000000000400631 <sigusr1_handler+17>:     5d      pop    %rbp
 => 0x0000000000400632 <sigusr1_handler+18>:     c3      retq
 1: $eflags = [ PF ZF TF IF ]
 (gdb)
 <signal handler called>
 => 0x0000003b36a358f0 <__restore_rt+0>: 48 c7 c0 0f 00 00 00    mov    $0xf,%rax
 1: $eflags = [ PF ZF TF IF ]
 (gdb) si
 <signal handler called>
 => 0x0000003b36a358f7 <__restore_rt+7>: 0f 05   syscall
 1: $eflags = [ PF ZF TF IF ]
 (gdb)
 main () at si-handler.c:50
 50        global = 0; /* set break here */
 => 0x000000000040066b <main+9>: c7 05 cb 09 20 00 00 00 00 00   movl   $0x0,0x2009cb(%rip)        # 0x601040 <global>
 1: $eflags = [ PF ZF IF ]
 (gdb)

The bug doesn't happen if we instead PTRACE_CONTINUE into the signal
handler -- e.g., set a breakpoint in the handler, queue a signal, and
"continue".

gdb/testsuite/
2014-10-28  Pedro Alves  <palves@redhat.com>

	PR gdb/17511
	* gdb.base/sigstep.c (handler): Add a few more writes to 'done'.
	* gdb.base/sigstep.exp (other_handler_location): New global.
	(advance): Support stepping into the signal handler, and running
	commands while in the handler.
	(in_handler_map): New global.
	(top level): In the advance test, add combinations for getting
	into the handler with stepping commands, and for running commands
	in the handler.  Add comment descripting the advancei tests.
PR gdb/12623: non-stop crashes inferior, PC adjustment and 1-byte insns

TL;DR - if we step an instruction that is as long as
decr_pc_after_break (1-byte on x86) right after removing the
breakpoint at PC, in non-stop mode, adjust_pc_after_break adjusts the
PC, but it shouldn't.

In non-stop mode, when a breakpoint is removed, it is moved to the
"moribund locations" list.  This is because other threads that are
running may have tripped on that breakpoint as well, and we haven't
heard about it.  When a trap is reported, we check if perhaps it was
such a deleted breakpoint that caused the trap.  If so, we also need
to adjust the PC (decr_pc_after_break).

Now, say that, on x86:

 - a breakpoint was placed at an address where we have an instruction
of the same length as decr_pc_after_break on this arch (1 on x86).

 - the breakpoint is removed, and thus put on the moribund locations
   list.

 - the thread is single-stepped.

As there's no breakpoint inserted at PC anymore, the single-step
actually executes the 1-byte instruction normally.  GDB should _not_
adjust the PC for the resulting SIGTRAP.  But, adjust_pc_after_break
confuses the step SIGTRAP reported for this single-step as being a
SIGTRAP for the moribund location of the breakpoint that used to be at
the previous PC, and so infrun applies the decr_pc_after_break
adjustment incorrectly.

The confusion comes from the special case mentioned in the comment:

 static void
 adjust_pc_after_break (struct execution_control_state *ecs)
 {
 ...
	  As a special case, we could have hardware single-stepped a
	  software breakpoint.  In this case (prev_pc == breakpoint_pc),
	  we also need to back up to the breakpoint address.  */

       if (thread_has_single_step_breakpoints_set (ecs->event_thread)
	   || !ptid_equal (ecs->ptid, inferior_ptid)
	   || !currently_stepping (ecs->event_thread)
	   || (ecs->event_thread->stepped_breakpoint
	       && ecs->event_thread->prev_pc == breakpoint_pc))
	 regcache_write_pc (regcache, breakpoint_pc);

The condition that incorrectly triggers is the
"ecs->event_thread->prev_pc == breakpoint_pc" one.

Afterwards, the next resume resume re-executes an instruction that had
already executed, which if you're lucky, results in the inferior
crashing.  If you're unlucky, you'll get silent bad behavior...

The fix is to remember that we stepped a breakpoint.  Turns out the
only case we step a breakpoint instruction today isn't covered by the
testsuite.  It's the case of a 'handle nostop" signal arriving while a
step is in progress _and_ we have a software watchpoint, which forces
always single-stepping.  This commit extends sigstep.exp to cover
that, and adds a new test for the adjust_pc_after_break issue.

Tested on x86_64 Fedora 20, native and gdbserver.

gdb/
2014-10-28  Pedro Alves  <palves@redhat.com>

	PR gdb/12623
	* gdbthread.h (struct thread_info) <stepped_breakpoint>: New
	field.
	* infrun.c (resume) <stepping breakpoint instruction>: Set the
	thread's stepped_breakpoint field.  Skip if reverse debugging.
	Add comment.
	(init_thread_stepping_state, handle_signal_stop): Clear the
	thread's stepped_breakpoint field.

gdb/testsuite/
2014-10-28  Pedro Alves  <palves@redhat.com>

	PR gdb/12623
	* gdb.base/sigstep.c (no_handler): New global.
	(main): If 'no_handler is true, set the signal handlers to
	SIG_IGN.
	* gdb.base/sigstep.exp (breakpoint_over_handler): Add
	with_sw_watch and no_handler parameters.  Handle them.
	(top level) <stepping over handler when stopped at a breakpoint
	test>: Add a test axis for testing with a software watchpoint, and
	another for testing with the signal handler set to SIG_IGN.
	* gdb.base/step-sw-breakpoint-adjust-pc.c: New file.
	* gdb.base/step-sw-breakpoint-adjust-pc.exp: New file.
daily update
Fix skipping stack protector on arm

This patch fixes the bug in my patch skipping stack protector
https://www.sourceware.org/ml/gdb-patches/2010-12/msg00110.html

In my skipping stack protector patch, I misunderstood the constant vs.
immediate on instruction encodings, and treated immediate as constant
by mistake.  The instruction 'ldr Rd, [PC, #immed]' loads the
address of __stack_chk_guard to Rd, and #immed is an offset from PC.
We should get the __stack_chk_guard from *(pc + #immed).

As a result of this mistake, arm_analyze_load_stack_chk_guard returns
the wrong address of __stack_chk_guard, and the symbol
__stack_chk_guard can't be found.  However, we continue to match the
following instructions when symbol isn't found, so the code still
works.  In other words, the code just matches the instruction pattern
without checking __stack_chk_guard symbol correctly.

Joel's patch <https://sourceware.org/ml/gdb-patches/2014-10/msg00605.html>
makes the heuristics stricter that we stop matching instructions if
symbol __stack_chk_guard isn't found.  Then the bug is exposed.  This
patch is to correct the load address computation for ldr instruction,
and it fixes some fails in gdb.mi/gdb792.exp on armv4t both arm and
thumb mode.

Regression tested on arm-linux-gnueabi target with
{armv4t, armv7-a} x {marm, mthumb} x {-fstack-protector,-fno-stack-protector}

gdb:

2014-10-29  Yao Qi  <yao@codesourcery.com>

	* arm-tdep.c (arm_analyze_load_stack_chk_guard): Compute the
	loaded address correctly of ldr instruction.
ARM: stricter __stack_chk_guard check during prologue analysis

We are trying to insert a breakpoint on line 4 for the following
Ada code.

  3 procedure STR is
  4    XX : String (1 .. Blocks.Sz) := (others => 'X'); -- STOP
  5    K : Integer;
  6 begin
  7    K := 13;

The code generated on ARM (-march=armv7-m) starts like this:

    (gdb) disass str'address
    Dump of assembler code for function _ada_str:
       --# Line str.adb:3
       0x08000014 <+0>:     push    {r4, r7, lr}
       0x08000016 <+2>:     sub     sp, #28
       0x08000018 <+4>:     add     r7, sp, #0
       0x0800001a <+6>:     mov     r3, sp
       0x0800001c <+8>:     mov     r4, r3
       --# Line str.adb:4
       0x0800001e <+10>:    ldr     r3, [pc, #84]   ; (0x8000074 <_ada_str+96>)
       0x08000020 <+12>:    ldr     r3, [r3, #0]
       0x08000022 <+14>:    str     r3, [r7, #20]
       0x08000024 <+16>:    ldr     r3, [r7, #20]
       [...]

When computing the address related to str.adb:4, GDB correctly
resolves it to 0x0800001e first, but then considers the next
3 instructions as being part of the prologue because it thinks
they are part of stack-protector code. As a result, instead
of inserting the breakpoint at line 4, it skips those instruction
and consequently the rest of the instructions until the start
of the next line, which his line 7.

The stack-protector code is expected to start like this...

        ldr     Rn, .Label
        ....
        .Lable:
        .word   __stack_chk_guard

... but the implementation actually accepts a sequence where
the ldr location points to an address for which there is no symbol.
It only aborts if the address points to a symbol which is not
__stack_chk_guard.

Since the __stack_chk_guard symbol is always expected to exist
when used (it lives in .dynsym), this patch fixes the issue by
requiring that the ldr gets the address of the __stack_chk_guard
symbol. If the address could not be resolved, then it rejects
the sequence as being stack-protector code.

gdb/ChangeLog:

        * arm-tdep.c (arm_skip_stack_protector): Return early if
        address loaded by first "ldr" instruction does not have
        a corresponding minimal symbol.  Update comment.

Tested on arm-eabi using AdaCore's testsuite.
Tested on arm-linux-gnueabi by Yao as well.
Close the file in fileio.exp test

I see the following fail in fileio.exp on mingw32 host gdb,

rename 1: ret = -1, errno = 13^M
^M
Breakpoint 2, stop () at fileio.c:76^M
76      static void stop () {}^M
(gdb) FAIL: gdb.base/fileio.exp: Rename a file

the test fails to rename a file which is not expected.  The previous
test test_write doesn't close the file, so the rename fails as a
result on Windows.  This patch fixes it by closing file in test_write,
and the fail goes away.

rename 1: ret = 0, errno = 0 OK^M
^M
Breakpoint 2, stop () at fileio.c:76^M
76      static void stop () {}^M
(gdb) PASS: gdb.base/fileio.exp: Rename a file

gdb/testsuite:

2014-10-29  Yao Qi  <yao@codesourcery.com>

	* gdb.base/fileio.c (test_write): Close the file.
Prepare directory in case test_system fails

In gdb.base/fileio.c, some functions may depend on others.  For
example, test_rename renames a file to one directory which is created
in test_system.  That is means, if test_system fails, test_rename
fails too, which is not a good practise, IMO.

In test_system, system ("mkdir -p XX") is used to create directories
needed for test_rename.  In this patch, we use dejagnu remote_exec
proc to create these directories on host.

In my gdb testing, mingw32 host and arm-none-eabi target, system
("mkdir -p XX") doesn't work properly (this issue can be addressed
separately), and this patch fixes the following fails.

FAIL: gdb.base/fileio.exp: Renaming a directory to a non-empty directory returns ENOTEMPTY or EEXIST
FAIL: gdb.base/fileio.exp: Unlink a file
FAIL: gdb.base/fileio.exp: Unlinking a file in a directory w/o write access returns EACCES

gdb/testsuite:

2014-10-29  Yao Qi  <yao@codesourcery.com>

	* gdb.base/fileio.exp: Make directories on host.
PR tui/16138, PR tui/17519, and misc failures to initialize the terminal

PR tui/16138 is about failure to initialize curses resulting in GDB
exiting instead of throwing an error.  E.g.:

 $ TERM=foo gdb
 (gdb) layout asm
 Error opening terminal: foo.
 $

The problem is that we're calling initscr to initialize the screen.
As mentioned in
http://pubs.opengroup.org/onlinepubs/7908799/xcurses/initscr.html:

 If errors occur, initscr() writes an appropriate error message to
 standard error and exits.
                    ^^^^^

Instead, we should use newterm:

 "A program that needs an indication of error conditions, so it can
 continue to run in a line-oriented mode if the terminal cannot support
 a screen-oriented program, would also use this function."

After the patch:

 $ TERM=foo gdb -q -nx
 (gdb) layout asm
 Cannot enable the TUI: error opening terminal [TERM=foo]
 (gdb)

And then PR tui/17519 is about GDB not validating whether the terminal
has the necessary capabilities when enabling the TUI.  If one tries to
enable the TUI with TERM=dumb (and e.g., from a shell within emacs),
GDB ends up with a clear screen, the cursor is placed at the
bottom/right corner of the screen, there's no prompt, typing shows no
echo, and there's no indication of what's going on.  c-x,a gets you
out of the TUI, but it's completely non-obvious.

After the patch, we get:

 $ TERM=dumb gdb -q -nx
 (gdb) layout asm
 Cannot enable the TUI: terminal doesn't support cursor addressing [TERM=dumb]
 (gdb)

While at it, I've moved all the tui_allowed_p validation to
tui_enable, and expanded the error messages.  Previously we'd get:

 $ gdb -q -nx -i=mi
 (gdb)
 layout asm
 &"layout asm\n"
 &"TUI mode not allowed\n"
 ^error,msg="TUI mode not allowed"

and:

 $ gdb -q -nx -ex "layout asm" > foo
 TUI mode not allowed

While now we get:

 $ gdb -q -nx -i=mi
 (gdb)
 layout asm
 &"layout asm\n"
 &"Cannot enable the TUI when the interpreter is 'mi'\n"
 ^error,msg="Cannot enable the TUI when the interpreter is 'mi'"
 (gdb)

and:

 $ gdb -q -nx -ex "layout asm" > foo
 Cannot enable the TUI when output is not a terminal

Tested on x86_64 Fedora 20.

gdb/
2014-10-29  Pedro Alves  <palves@redhat.com>

	PR tui/16138
	PR tui/17519
	* tui/tui-interp.c (tui_is_toplevel): Delete global.
	(tui_allowed_p): Delete function.
	* tui/tui.c: Include "interps.h".
	(tui_enable): Don't use tui_allowed_p.  Error out here with
	detailed error messages if the TUI is the top level interpreter,
	or if output is not a terminal.  Use newterm instead of initscr,
	and error out if initializing the terminal fails.  Also error out if
	the terminal doesn't support cursor addressing.
	* tui/tui.h (tui_allowed_p): Delete declaration.
TUI: don't let exceptions escape while handling readline key bindings

I noticed that with:

 $ TERM=dumb ./gdb -q -nx
 <c-x,a>
 Cannot enable the TUI: terminal doesn't support cursor addressing [TERM=dumb]
 (gdb)

The next key the user types is silently eaten.

The problem is that we're throwing an exception while in a readline
callback that isn't prepared for that:

(top-gdb) bt
#0  tui_enable () at /home/pedro/gdb/mygit/build/../src/gdb/tui/tui.c:388
#1  0x000000000051f47b in tui_rl_switch_mode (notused1=1, notused2=1) at /home/pedro/gdb/mygit/build/../src/gdb/tui/tui.c:101
#2  0x0000000000768d6f in _rl_dispatch_subseq (key=1, map=0xd069c0 <emacs_ctlx_keymap>, got_subseq=0) at /home/pedro/gdb/mygit/build/../src/readline/readline.c:774
#3  0x0000000000768acb in _rl_dispatch_callback (cxt=0x1ce6190) at /home/pedro/gdb/mygit/build/../src/readline/readline.c:686
#4  0x000000000078120b in rl_callback_read_char () at /home/pedro/gdb/mygit/build/../src/readline/callback.c:170
#5  0x0000000000619445 in rl_callback_read_char_wrapper (client_data=0x0) at /home/pedro/gdb/mygit/build/../src/gdb/event-top.c:166
#6  0x000000000061981b in stdin_event_handler (error=0, client_data=0x0) at /home/pedro/gdb/mygit/build/../src/gdb/event-top.c:372
#7  0x000000000061840e in handle_file_event (data=...) at /home/pedro/gdb/mygit/build/../src/gdb/event-loop.c:762
#8  0x00000000006178f5 in process_event () at /home/pedro/gdb/mygit/build/../src/gdb/event-loop.c:339
#9  0x00000000006179bc in gdb_do_one_event () at /home/pedro/gdb/mygit/build/../src/gdb/event-loop.c:403
#10 0x0000000000617a0c in start_event_loop () at /home/pedro/gdb/mygit/build/../src/gdb/event-loop.c:428

Here, in _rl_dispatch_subseq:

769
770               rl_executing_keymap = map;
771
772               rl_dispatching = 1;
773               RL_SETSTATE(RL_STATE_DISPATCHING);
774               (*map[key].function)(rl_numeric_arg * rl_arg_sign, key);
775               RL_UNSETSTATE(RL_STATE_DISPATCHING);
776               rl_dispatching = 0;
777
778               /* If we have input pending, then the last command was a prefix
779                  command.  Don't change the state of rl_last_func.  Otherwise,

GDB is called from line 774, but longjmp'ing at that point leaves
rl_dispatching and RL_STATE_DISPATCHING set.

Fix this by wrapping tui_rl_switch_mode in a TRY_CATCH.

gdb/
2014-10-29  Pedro Alves  <palves@redhat.com>

	* tui/tui.c (tui_rl_switch_mode): Wrap tui_enable/tui_disable in
	TRY_CATCH.