Q:

How can I debug nginx further than the error log? 

I'm currently receiving a fairly large HTTP flood right now, and it's causing my nginx reverse proxy to produce a 502 Bad Gateway. 
I have a frontend server running nginx as a proxy to my backend server, but it's just getting a bunch of connect() failed (110: Connection timed out) while connecting to upstream errors.  Tons of them.  If I bypass the proxy server to connect to the backend, I can run the site just fine, so I know it's in the reverse proxy somewhere.  However, I have no idea how to determine why it's timing out. 
Any help? 
running nginx 1.2.3 on CentOS 6.2

A:

It doesn't get much more pedantic than this unless you want to put in dtrace probes:

Set debug log level: /etc/nginx/nginx.conf:
...
http {
        ...
        error_log /var/log/nginx/error.log debug; # todo testing remove me not for production use
        ...
}

Setup tcpdump in another window:
tcpdump not port 22 -vvv -s0 -q -XXX

Monitor log files in yet another window:
tail -f /var/log/nginx/*

Startup nginx interactively with strace:
# top of /etc/nginx/nginx.conf:

daemon off; # todo testing remove me not for production use

And then
 $ strace nginx 

Further debugging can be had with an nginx compiled with --with-debug.   Check it by running:
    nginx -V 2>&1 | grep -- '--with-debug' # no output if not debug

Another good module not compiled by default is: HttpStubStatusModule.   In all likelihood, any decent setup will require a custom-compiled nginx (highly-recommend packaging using distro's packaging tools). 
Most of these are unsuitable for production use, look at compiling nginx with gperf if you need more stats. 

A:

I'm assuming you've already jacked your Nginx error logging level up to debug.  If not, start there. 
Your best bet is probably going to be using strace to view the system calls being made by Nginx.  In particular, you'll want to pay attention to connect() calls, and keep an eye on the return codes of these (man 2 connect can be your friend here). 
Once you have that information, you can better make an educated guess about whether the issue is confined to your frontend proxy, or has something to do with the interactions between the proxy and backend application server. 

A:

Looks like you are debugging a high traffic site. 
Use debug with debug_connection directive so nginx error log will show debug logs from your IP only.  
Once you start seeing some useful error logs rather than activating debug option for entire nginx config, add a separate error_log /path/to/some/file/ debug; directive in location {..} block responsible for reverse_proxy connection. 
This way you will be able to isolate debug error log from your IP only. 
Try to relate it with request you are making (from your browser). 
For example, please check: https://easyengine.io/tutorials/nginx/debugging/
A level ahead, you can use Nginx's HttpEchoModule 

