Repository: istio/fortio
Pull request #200 (2018-03-11 18:49:59)

curl -L mode; move generic code out of uihandler; more tests; etc...
- moved generic code from ui to fhttp/
- added `-L` and `FollowRedirects` options (meant for `fortio curl -L url > file`). -L is shortcut for both follow redirect and using the stdclient (decoding the data part, vs raw output from fast client). So `fortio curl -L ...` can be used for downloads.
- `FetchURL` and `Fetch` helper functions wrapping 1 time use client to help write 1 liner readiness
- more tests
- added go version in `fortio version` output (`version.Long()`)
- changed `-httpreqtimeout` flag to just `-timeout`
- expose `fhttp.RedirectToHTTPS`, `NewHTMLEscapeWriter`, `SetupPPROF`, `FetcherHandler` proxy, `LogRequest`, `OnBehalfOf` to the public fhttp library api.
- solved README.md wrapping of fortio command help text using new `release/updateFlags.sh`


codecov[bot]: # [Codecov](https://codecov.io/gh/istio/fortio/pull/200?src=pr&el=h1) Report
> Merging [#200](https://codecov.io/gh/istio/fortio/pull/200?src=pr&el=desc) into [master](https://codecov.io/gh/istio/fortio/commit/bdf66397a30eabf53ae822321b21a20d70d6383b?src=pr&el=desc) will **decrease** coverage by `1.7%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/istio/fortio/pull/200/graphs/tree.svg?height=150&width=650&token=6HLno2jpvU&src=pr)](https://codecov.io/gh/istio/fortio/pull/200?src=pr&el=tree)

```diff
@@           Coverage Diff            @@
##           master    #200     +/-   ##
========================================
- Coverage    89.9%   88.2%   -1.7%     
========================================
  Files          10      10             
  Lines        1670    1702     +32     
========================================
  Hits         1501    1501             
- Misses        112     144     +32     
  Partials       57      57
```


| [Impacted Files](https://codecov.io/gh/istio/fortio/pull/200?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [fhttp/http\_utils.go](https://codecov.io/gh/istio/fortio/pull/200/diff?src=pr&el=tree#diff-Zmh0dHAvaHR0cF91dGlscy5nbw==) | `97.3% <0%> (-2.7%)` | :arrow_down: |
| [fhttp/http\_server.go](https://codecov.io/gh/istio/fortio/pull/200/diff?src=pr&el=tree#diff-Zmh0dHAvaHR0cF9zZXJ2ZXIuZ28=) | `69.7% <0%> (-12.4%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/istio/fortio/pull/200?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/istio/fortio/pull/200?src=pr&el=footer). Last update [bdf6639...84c222d](https://codecov.io/gh/istio/fortio/pull/200?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ldemailly: @mattdelco the `fortio curl -L ...` is for you (and the fhttp.FetchURL() for some of our liveness/readiness tests)
mattdelco: Thanks, it looks alright to me (though I don't consider myself very knowledgeable about fortio or golang).  I have been a bit apprehensive about relying on stdout (vs an explicit file output parameter), but so far things seem okay (almost everything else is using stderr, either directly or indirectly via the logging library).
ldemailly: any unix-y program worth its salt will not mix stdout and stderr :-) if it produces something to be saved it will be on stdout while logging goes to stderr

Patch:
diff --git a/README.md b/README.md
index 0a1251d..d889028 100644
--- a/README.md
+++ b/README.md
@@ -28,7 +28,7 @@ docker run istio/fortio load http://www.google.com/ # For a test run
 Or download the binary distribution, for instance:
 
 ```shell
-curl -L https://github.com/istio/fortio/releases/download/v0.7.3/fortio-linux_x64-0.7.3.tgz \
+curl -L https://github.com/istio/fortio/releases/download/v0.8.0/fortio-linux_x64-0.8.0.tgz \
  | sudo tar -C / -xvzpf -
 ```
 
@@ -39,111 +39,112 @@ You can get a preview of the reporting/graphing UI at https://fortio.istio.io/
 ## Command line arguments
 
 Fortio can be an http or grpc load generator, gathering statistics using the `load` subcommand, or start simple http and grpc ping servers, as well as a basic web UI, result graphing and https redirector, with the `server` command or issue grpc ping messages using the `grpcping` command. It can also fetch a single URL's for debugging when using the `curl` command (or the `-curl` flag to the load command). You can run just the redirector with `redirect`. Lastly if you saved JSON results (using the web UI or directly from the command line), you can browse and graph those results using the `report` command.
-
+<!-- use release/updateFlags.sh to update this section -->
 ```
-Φορτίο 0.7.3 usage:
+Φορτίο 0.8.0 usage:
 	fortio command [flags] target
-where command is one of: load (load testing), server (starts grpc ping and
-http echo/ui/redirect servers), grpcping (grpc client), report (report only UI
-server), redirect (redirect only server), or curl (single URL debug).
-where target is a url (http load tests) or host:port (grpc health test)
-and flags are:
+where command is one of: load (load testing), server (starts grpc ping and http
+echo/ui/redirect servers), grpcping (grpc client), report (report only UI
+server), redirect (redirect only server), or curl (single URL debug).  where
+target is a url (http load tests) or host:port (grpc health test) and flags are:
   -H value
-    	Additional Header(s)
+	Additional Header(s)
+  -L	Follow redirects (implies -std-client) - do not use for load test
   -a	Automatically save JSON result with filename based on labels & timestamp
   -allow-initial-errors
-    	Allow and don't abort on initial warmup errors
+	Allow and don't abort on initial warmup errors
   -base-url string
-    	base URL used as prefix for data/index.tsv generation. (when empty, the
-    	url from the first request is used)
+	base URL used as prefix for data/index.tsv generation. (when empty, the
+	url from the first request is used)
   -c int
-    	Number of connections/goroutine/threads (default 4)
+	Number of connections/goroutine/threads (default 4)
   -compression
-    	Enable http compression
+	Enable http compression
   -curl
-    	Just fetch the content once
+	Just fetch the content once
   -data-dir string
-    	Directory where JSON results are stored/read (default ".")
+	Directory where JSON results are stored/read (default ".")
   -echo-debug-path string
-    	http echo server URI for debug, empty turns off that part (more secure)
-    	(default "/debug")
+	http echo server URI for debug, empty turns off that part (more secure)
+	(default "/debug")
   -gomaxprocs int
-    	Setting for runtime.GOMAXPROCS, <1 doesn't change the default
+	Setting for runtime.GOMAXPROCS, <1 doesn't change the default
   -grpc
-    	Use GRPC (health check) for load testing
+	Use GRPC (health check) for load testing
   -grpc-port string
-    	grpc server port. Can be in the form of host:port, ip:port or port.
-    	(default "8079")
+	grpc server port. Can be in the form of host:port, ip:port or port.
+	(default "8079")
   -grpc-secure
-    	Use secure transport (tls) for GRPC
+	Use secure transport (tls) for GRPC
   -halfclose
-    	When not keepalive, whether to half close the connection (only for fast
-    	http)
+	When not keepalive, whether to half close the connection (only for fast
+	http)
   -health
-    	grpc ping client mode: use health instead of ping
+	grpc ping client mode: use health instead of ping
   -healthservice string
-    	which service string to pass to health check
+	which service string to pass to health check
   -http-port string
-    	http echo server port. Can be in the form of host:port, ip:port or port.
-    	(default "8080")
+	http echo server port. Can be in the form of host:port, ip:port or port.
+	(default "8080")
   -http1.0
-    	Use http1.0 (instead of http 1.1)
+	Use http1.0 (instead of http 1.1)
   -httpbufferkb int
-    	Size of the buffer (max data size) for the optimized http client in kbytes
-    	(default 128)
+	Size of the buffer (max data size) for the optimized http client in
+	kbytes (default 128)
   -httpccch
-    	Check for Connection: Close Header
-  -httpreqtimeout duration
-    	Http request timeout value (default 15s)
+	Check for Connection: Close Header
   -https-insecure
-    	Do not verify certs in https connections
+	Do not verify certs in https connections
   -json string
-    	Json output to provided file or '-' for stdout (empty = no json output,
-    	unless -a is used)
+	Json output to provided file or '-' for stdout (empty = no json output,
+	unless -a is used)
   -keepalive
-    	Keep connection alive (only for fast http 1.1) (default true)
+	Keep connection alive (only for fast http 1.1) (default true)
   -labels string
-    	Additional config data/labels to add to the resulting JSON, defaults to
-    	target URL and hostname
+	Additional config data/labels to add to the resulting JSON, defaults to
+	target URL and hostname
   -logcaller
-    	Logs filename and line number of callers to log (default true)
+	Logs filename and line number of callers to log (default true)
   -loglevel value
-    	loglevel, one of [Debug Verbose Info Warning Error Critical Fatal]
-      (default Info)
+	loglevel, one of [Debug Verbose Info Warning Error Critical Fatal]
+	(default Info)
   -logprefix string
-    	Prefix to log lines before logged messages (default "> ")
+	Prefix to log lines before logged messages (default "> ")
   -maxpayloadsizekb int
-    	MaxPayloadSizeKb is the maximum size of payload to be generated by the EchoHandler size= argument (In kilobytes defaults to 256)
+	MaxPayloadSize is the maximum size of payload to be generated by the
+	EchoHandler size= argument. In Kbytes. (default 256)
   -n int
-    	Run for exactly this number of calls instead of duration. Default (0) is
-    	to use duration (-t). Default is 1 when used as grpc ping count.
+	Run for exactly this number of calls instead of duration. Default (0) is
+	to use duration (-t). Default is 1 when used as grpc ping count.
   -p string
-    	List of pXX to calculate (default "50,75,90,99,99.9")
+	List of pXX to calculate (default "50,75,90,99,99.9")
   -payload string
-    	Payload string to send along
+	Payload string to send along
   -profile string
-    	write .cpu and .mem profiles to file
+	write .cpu and .mem profiles to file
   -qps float
-    	Queries Per Seconds or 0 for no wait/max qps (default 8)
+	Queries Per Seconds or 0 for no wait/max qps (default 8)
   -quiet
-    	Quiet mode: sets the loglevel to Error and reduces the output.
+	Quiet mode: sets the loglevel to Error and reduces the output.
   -r float
-    	Resolution of the histogram lowest buckets in seconds (default 0.001)
+	Resolution of the histogram lowest buckets in seconds (default 0.001)
   -redirect-port string
-    	Redirect all incoming traffic to https URL (need ingress to work properly)
-    	Can take the form of host:port, ip:port, port or "disabled" to disable
-    	the feature. (default "8081")
+	Redirect all incoming traffic to https URL (need ingress to work
+	properly). Can be in the form of host:port, ip:port, port or "disabled"
+	to disable the feature. (default "8081")
   -static-dir string
-    	Absolute path to the dir containing the static files dir
+	Absolute path to the dir containing the static files dir
   -stdclient
-    	Use the slower net/http standard client (works for TLS)
+	Use the slower net/http standard client (works for TLS)
   -sync string
-    	index.tsv or s3/gcs bucket xml URL to fetch at startup for server modes.
+	index.tsv or s3/gcs bucket xml URL to fetch at startup for server modes.
   -t duration
-    	How long to run the test or 0 to run until ^C (default 5s)
+	How long to run the test or 0 to run until ^C (default 5s)
+  -timeout duration
+	Connection and read timeout value (for http) (default 15s)
   -ui-path string
-    	http server URI for UI, empty turns off that part (more secure)
-    	(default "/fortio/")
+	http server URI for UI, empty turns off that part (more secure) (default
+	"/fortio/")
 ```
 
 ## Example use and output
@@ -151,11 +152,12 @@ and flags are:
 * Start the internal servers:
 ```
 $ fortio server &
-Https redirector running on :8081
-UI starting - visit:
+Fortio 0.8.0 grpc 'ping' server listening on [::]:8079
+Fortio 0.8.0 https redirector server listening on [::]:8081
+Fortio 0.8.0 echo server listening on [::]:8080
+UI started - visit:
 http://localhost:8080/fortio/   (or any host/ip reachable on this server)
-Fortio 0.7.3 grpc ping server listening on port :8079
-Fortio 0.7.3 echo server listening on port :8080
+21:45:23 I fortio_main.go:195> All fortio 0.8.0 buildinfo go1.10 servers started!
 ```
 
 * By default, Fortio's web/echo servers listen on port 8080 on all interfaces.
@@ -165,8 +167,8 @@ $ fortio server -http-port 10.10.10.10:8088
 UI starting - visit:
 http://10.10.10.10:8088/fortio/
 Https redirector running on :8081
-Fortio 0.7.3 grpc ping server listening on port :8079
-Fortio 0.7.3 echo server listening on port 10.10.10.10:8088
+Fortio 0.8.0 grpc ping server listening on port :8079
+Fortio 0.8.0 echo server listening on port 10.10.10.10:8088
 ```
 * Simple grpc ping:
 ```
@@ -236,14 +238,14 @@ Content-Type: text/plain; charset=UTF-8
 Date: Mon, 08 Jan 2018 22:26:26 GMT
 Content-Length: 230
 
-Φορτίο version 0.7.3 echo debug server up for 39s on ldemailly-macbookpro - request from [::1]:65055
+Φορτίο version 0.8.0 echo debug server up for 39s on ldemailly-macbookpro - request from [::1]:65055
 
 GET /debug HTTP/1.1
 
 headers:
 
 Host: localhost:8080
-User-Agent: istio/fortio-0.7.3
+User-Agent: istio/fortio-0.8.0
 Foo: Bar
 
 body:
diff --git a/fhttp/http_client.go b/fhttp/http_client.go
index 02d6d61..763f701 100644
--- a/fhttp/http_client.go
+++ b/fhttp/http_client.go
@@ -128,6 +128,7 @@ type HTTPOptions struct {
 	DisableKeepAlive  bool // so default is keep alive
 	AllowHalfClose    bool // if not keepalive, whether to half close after request
 	Insecure          bool // do not verify certs for https
+	FollowRedirects   bool // For the Std Client only: follow redirects.
 	initDone          bool
 	https             bool // whether URLSchemeCheck determined this was an https:// call or not
 	// ExtraHeaders to be added to each request.
@@ -314,16 +315,38 @@ func NewStdClient(o *HTTPOptions) *Client {
 		&http.Client{
 			Timeout:   o.HTTPReqTimeOut,
 			Transport: &tr,
-			// Lets us see the raw response instead of auto following redirects.
-			CheckRedirect: func(req *http.Request, via []*http.Request) error {
-				return http.ErrUseLastResponse
-			},
 		},
 		&tr,
 	}
+	if !o.FollowRedirects {
+		// Lets us see the raw response instead of auto following redirects.
+		client.client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
+			return http.ErrUseLastResponse
+		}
+	}
 	return &client
 }
 
+// FetchURL fetches the data at the given url using the standard client and default options.
+// Returns the http status code (http.StatusOK == 200 for success) and the data.
+// To be used only for single fetches or when performance doesn't matter as the client is closed at the end.
+func FetchURL(url string) (int, []byte) {
+	o := NewHTTPOptions(url)
+	// Maximize chances of getting the data back, vs the raw payload like the fast client
+	o.DisableFastClient = true
+	o.FollowRedirects = true
+	return Fetch(o)
+}
+
+// Fetch creates a client an performs a fetch according to the http options passed in.
+// To be used only for single fetches or when performance doesn't matter as the client is closed at the end.
+func Fetch(httpOptions *HTTPOptions) (int, []byte) {
+	cli := NewClient(httpOptions)
+	code, data, _ := cli.Fetch()
+	cli.Close()
+	return code, data
+}
+
 // FastClient is a fast, lockfree single purpose http 1.0/1.1 client.
 type FastClient struct {
 	buffer       []byte
diff --git a/fhttp/http_server.go b/fhttp/http_server.go
index e005792..f526a29 100644
--- a/fhttp/http_server.go
+++ b/fhttp/http_server.go
@@ -24,10 +24,11 @@ import (
 	"os"
 	"sort"
 	"strconv"
+	"strings"
 	"sync/atomic"
 	"time"
-	// get /debug/pprof endpoints on default server - make sure report only doesn't have it
-	_ "net/http/pprof"
+	// get /debug/pprof endpoints on a mux through SetupPPROF
+	"net/http/pprof"
 
 	"istio.io/fortio/fnet"
 	"istio.io/fortio/log"
@@ -50,7 +51,9 @@ var (
 
 // EchoHandler is an http server handler echoing back the input.
 func EchoHandler(w http.ResponseWriter, r *http.Request) {
-	log.LogVf("%v %v %v %v", r.Method, r.URL, r.Proto, r.RemoteAddr)
+	if log.LogVerbose() {
+		LogRequest(r, "Echo") // will also print headers
+	}
 	data, err := ioutil.ReadAll(r.Body) // must be done before calling FormValue
 	if err != nil {
 		log.Errf("Error reading %v", err)
@@ -74,11 +77,6 @@ func EchoHandler(w http.ResponseWriter, r *http.Request) {
 		// TODO: this easily lead to contention - use 'thread local'
 		rqNum := atomic.AddInt64(&EchoRequests, 1)
 		log.Debugf("Request # %v", rqNum)
-		for name, headers := range r.Header {
-			for _, h := range headers {
-				log.Debugf("Header %v: %v\n", name, h)
-			}
-		}
 	}
 	if r.FormValue("close") != "" {
 		log.Debugf("Adding Connection:close / will close socket")
@@ -238,7 +236,9 @@ environment:
 
 // DebugHandler returns debug/useful info to http client.
 func DebugHandler(w http.ResponseWriter, r *http.Request) {
-	log.LogVf("%v %v %v %v", r.Method, r.URL, r.Proto, r.RemoteAddr)
+	if log.LogVerbose() {
+		LogRequest(r, "Debug")
+	}
 	var buf bytes.Buffer
 	buf.WriteString("Φορτίο version ")
 	buf.WriteString(version.Long())
@@ -314,3 +314,86 @@ func Serve(port, debugPath string) (*http.ServeMux, *net.TCPAddr) {
 	mux.HandleFunc("/", EchoHandler)
 	return mux, addr
 }
+
+// -- formerly in ui handler
+
+// SetupPPROF add pprof to the mux (mirror the init() of http pprof).
+func SetupPPROF(mux *http.ServeMux) {
+	mux.HandleFunc("/debug/pprof/", LogAndCall("pprof:index", pprof.Index))
+	mux.HandleFunc("/debug/pprof/cmdline", LogAndCall("pprof:cmdline", pprof.Cmdline))
+	mux.HandleFunc("/debug/pprof/profile", LogAndCall("pprof:profile", pprof.Profile))
+	mux.HandleFunc("/debug/pprof/symbol", LogAndCall("pprof:symbol", pprof.Symbol))
+	mux.HandleFunc("/debug/pprof/trace", LogAndCall("pprof:trace", pprof.Trace))
+}
+
+// -- Fetch er (simple http proxy) --
+
+// FetcherHandler is the handler for the fetcher/proxy.
+func FetcherHandler(w http.ResponseWriter, r *http.Request) {
+	LogRequest(r, "Fetch (prefix stripped)")
+	hj, ok := w.(http.Hijacker)
+	if !ok {
+		log.Critf("hijacking not supported")
+		return
+	}
+	conn, _, err := hj.Hijack()
+	if err != nil {
+		log.Errf("hijacking error %v", err)
+		return
+	}
+	// Don't forget to close the connection:
+	defer conn.Close() // nolint: errcheck
+	// Stripped prefix gets replaced by ./ - sometimes...
+	url := strings.TrimPrefix(r.URL.String(), "./")
+	opts := NewHTTPOptions("http://" + url)
+	opts.HTTPReqTimeOut = 5 * time.Minute
+	OnBehalfOf(opts, r)
+	client := NewClient(opts)
+	if client == nil {
+		return // error logged already
+	}
+	_, data, _ := client.Fetch()
+	_, err = conn.Write(data)
+	if err != nil {
+		log.Errf("Error writing fetched data to %v: %v", r.RemoteAddr, err)
+	}
+	client.Close()
+}
+
+// -- Redirection to https feature --
+
+// RedirectToHTTPSHandler handler sends a redirect to same URL with https.
+func RedirectToHTTPSHandler(w http.ResponseWriter, r *http.Request) {
+	dest := "https://" + r.Host + r.URL.String()
+	LogRequest(r, "Redirecting to "+dest)
+	http.Redirect(w, r, dest, http.StatusSeeOther)
+}
+
+// RedirectToHTTPS Sets up a redirector to https on the given port.
+// (Do not create a loop, make sure this is addressed from an ingress)
+func RedirectToHTTPS(port string) *net.TCPAddr {
+	m, a := HTTPServer("https redirector", port)
+	m.HandleFunc("/", RedirectToHTTPSHandler)
+	return a
+}
+
+// LogRequest logs the incoming request, including headers when loglevel is verbose
+func LogRequest(r *http.Request, msg string) {
+	log.Infof("%s: %v %v %v %v (%s)", msg, r.Method, r.URL, r.Proto, r.RemoteAddr,
+		r.Header.Get("X-Forwarded-Proto"))
+	if log.LogVerbose() {
+		for name, headers := range r.Header {
+			for _, h := range headers {
+				log.LogVf("Header %v: %v\n", name, h)
+			}
+		}
+	}
+}
+
+// LogAndCall wrapps an HTTP handler to log the request first.
+func LogAndCall(msg string, hf http.HandlerFunc) http.HandlerFunc {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		LogRequest(r, msg)
+		hf(w, r)
+	})
+}
diff --git a/fhttp/http_utils.go b/fhttp/http_utils.go
index 99a64e0..df5d30b 100644
--- a/fhttp/http_utils.go
+++ b/fhttp/http_utils.go
@@ -16,6 +16,8 @@ package fhttp // import "istio.io/fortio/fhttp"
 
 import (
 	"fmt"
+	"html/template"
+	"io"
 	"math/rand"
 	"net/http"
 	"strconv"
@@ -415,3 +417,35 @@ func RoundDuration(d time.Duration) time.Duration {
 	r := int64(d+50*time.Millisecond) / tenthSec
 	return time.Duration(tenthSec * r)
 }
+
+// -- formerly in uihandler:
+
+// HTMLEscapeWriter is an io.Writer escaping the output for safe html inclusion.
+type HTMLEscapeWriter struct {
+	NextWriter io.Writer
+	Flusher    http.Flusher
+}
+
+func (w *HTMLEscapeWriter) Write(p []byte) (int, error) {
+	template.HTMLEscape(w.NextWriter, p)
+	if w.Flusher != nil {
+		w.Flusher.Flush()
+	}
+	return len(p), nil
+}
+
+// NewHTMLEscapeWriter creates a io.Writer that can safely output
+// to an http.ResponseWrite with HTMLEscape-ing.
+func NewHTMLEscapeWriter(w io.Writer) io.Writer {
+	flusher, ok := w.(http.Flusher)
+	if !ok {
+		log.Errf("expected writer %+v to be an http.ResponseWriter and thus a http.Flusher", w)
+		flusher = nil
+	}
+	return &HTMLEscapeWriter{NextWriter: w, Flusher: flusher}
+}
+
+// OnBehalfOf adds a header with the remote addr to an http options object.
+func OnBehalfOf(o *HTTPOptions, r *http.Request) {
+	_ = o.AddAndValidateExtraHeader("X-On-Behalf-Of: " + r.RemoteAddr)
+}
diff --git a/fortio_main.go b/fortio_main.go
index 24a6d43..d849cd3 100644
--- a/fortio_main.go
+++ b/fortio_main.go
@@ -81,7 +81,7 @@ var (
 	profileFlag        = flag.String("profile", "", "write .cpu and .mem profiles to file")
 	keepAliveFlag      = flag.Bool("keepalive", true, "Keep connection alive (only for fast http 1.1)")
 	halfCloseFlag      = flag.Bool("halfclose", false, "When not keepalive, whether to half close the connection (only for fast http)")
-	httpReqTimeoutFlag = flag.Duration("httpreqtimeout", fhttp.HTTPReqTimeOutDefaultValue, "Http request timeout value")
+	httpReqTimeoutFlag = flag.Duration("timeout", fhttp.HTTPReqTimeOutDefaultValue, "Connection and read timeout value (for http)")
 	stdClientFlag      = flag.Bool("stdclient", false, "Use the slower net/http standard client (works for TLS)")
 	http10Flag         = flag.Bool("http1.0", false, "Use http1.0 (instead of http 1.1)")
 	grpcFlag           = flag.Bool("grpc", false, "Use GRPC (health check) for load testing")
@@ -103,6 +103,7 @@ var (
 	headersFlags   flagList
 	defaultDataDir = "."
 
+	followRedirectsFlag    = flag.Bool("L", false, "Follow redirects (implies -std-client) - do not use for load test")
 	allowInitialErrorsFlag = flag.Bool("allow-initial-errors", false, "Allow and don't abort on initial warmup errors")
 	autoSaveFlag           = flag.Bool("a", false, "Automatically save JSON result with filename based on labels & timestamp")
 	redirectFlag           = flag.String("redirect-port", "8081", "Redirect all incoming traffic to https URL"+
@@ -170,18 +171,18 @@ func main() {
 		fortioLoad(*curlFlag, percList)
 	case "redirect":
 		isServer = true
-		ui.RedirectToHTTPS(*redirectFlag)
+		fhttp.RedirectToHTTPS(*redirectFlag)
 	case "report":
 		isServer = true
 		if *redirectFlag != "disabled" {
-			ui.RedirectToHTTPS(*redirectFlag)
+			fhttp.RedirectToHTTPS(*redirectFlag)
 		}
 		ui.Report(baseURL, *echoPortFlag, *staticDirFlag, *dataDirFlag)
 	case "server":
 		isServer = true
 		fgrpc.PingServer(*grpcPortFlag, fgrpc.DefaultHealthServiceName)
 		if *redirectFlag != "disabled" {
-			ui.RedirectToHTTPS(*redirectFlag)
+			fhttp.RedirectToHTTPS(*redirectFlag)
 		}
 		ui.Serve(baseURL, *echoPortFlag, *echoDbgPathFlag, *uiPathFlag, *staticDirFlag, *dataDirFlag, percList)
 	case "grpcping":
@@ -225,6 +226,10 @@ func fortioLoad(justCurl bool, percList []float64) {
 	httpOpts.Compression = *compressionFlag
 	httpOpts.HTTPReqTimeOut = *httpReqTimeoutFlag
 	httpOpts.Insecure = *httpsInsecureFlag
+	if *followRedirectsFlag {
+		httpOpts.FollowRedirects = true
+		httpOpts.DisableFastClient = true
+	}
 	if justCurl {
 		fetchURL(&httpOpts)
 		return
diff --git a/release/README.md b/release/README.md
index 5e9a5ff..4928f23 100644
--- a/release/README.md
+++ b/release/README.md
@@ -18,6 +18,8 @@
 
 - Once the release is deemed good/stable: move the git tag `latest_release` to the same as the release. Also push `latest_release` docker tag/image.
 
+- To update the command line flags in the ../README.md; go install the right version of fortio so it is in your path and run updateFlags.sh
+
 ## How to change the build image
 
 Update [../Dockerfile.build](../Dockerfile.build)
diff --git a/release/updateFlags.sh b/release/updateFlags.sh
new file mode 100755
index 0000000..bd99818
--- /dev/null
+++ b/release/updateFlags.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+# Extract fortio's help and rewrap it to 80 cols
+# fmt doesn't touch lines starting with . so we change the " -" to dot and back to keep
+# the option lines
+fortio 2>&1 | sed '$ d' | sed -e 's/^  -/./' | fmt -80 | sed -e 's/^\./  -/'
diff --git a/ui/uihandler.go b/ui/uihandler.go
index 7db115f..1668344 100644
--- a/ui/uihandler.go
+++ b/ui/uihandler.go
@@ -29,7 +29,6 @@ import (
 	"io"
 	"io/ioutil"
 	"net/http"
-	"net/http/pprof"
 	"net/url"
 	"os"
 	"path"
@@ -106,20 +105,6 @@ func getResourcesDir(override string) string {
 	return ""
 }
 
-// HTMLEscapeWriter is an io.Writer escaping the output for safe html inclusion.
-type HTMLEscapeWriter struct {
-	NextWriter io.Writer
-	Flusher    http.Flusher
-}
-
-func (w *HTMLEscapeWriter) Write(p []byte) (int, error) {
-	template.HTMLEscape(w.NextWriter, p)
-	if w.Flusher != nil {
-		w.Flusher.Flush()
-	}
-	return len(p), nil
-}
-
 // TODO: auto map from (Http)RunnerOptions to form generation and/or accept
 // JSON serialized options as input.
 
@@ -139,7 +124,7 @@ const (
 
 // Handler is the main UI handler creating the web forms and processing them.
 func Handler(w http.ResponseWriter, r *http.Request) {
-	LogRequest(r, "UI")
+	fhttp.LogRequest(r, "UI")
 	mode := menu
 	JSONOnly := false
 	DoSave := (r.FormValue("save") == "on")
@@ -189,7 +174,7 @@ func Handler(w http.ResponseWriter, r *http.Request) {
 		percList = defaultPercentileList
 	}
 	if !JSONOnly {
-		out = io.Writer(&HTMLEscapeWriter{NextWriter: w, Flusher: flusher})
+		out = fhttp.NewHTMLEscapeWriter(w)
 	}
 	n, _ := strconv.ParseInt(r.FormValue("n"), 10, 64) // nolint: gas
 	if strings.TrimSpace(url) == "" {
@@ -295,7 +280,7 @@ func Handler(w http.ResponseWriter, r *http.Request) {
 				log.Errf("Error adding custom headers: %v", err)
 			}
 		}
-		onBehalfOf(httpopts, r)
+		fhttp.OnBehalfOf(httpopts, r)
 		if !JSONOnly {
 			flusher.Flush()
 		}
@@ -407,7 +392,7 @@ func DataList() (dataList []string) {
 
 // BrowseHandler handles listing and rendering the JSON results.
 func BrowseHandler(w http.ResponseWriter, r *http.Request) {
-	LogRequest(r, "Browse")
+	fhttp.LogRequest(r, "Browse")
 	path := r.URL.Path
 	if (path != uiPath) && (path != (uiPath + "browse")) {
 		if strings.HasPrefix(path, "/fortio") {
@@ -443,23 +428,10 @@ func BrowseHandler(w http.ResponseWriter, r *http.Request) {
 	}
 }
 
-// LogRequest logs the incoming request, including headers when loglevel is verbose
-func LogRequest(r *http.Request, msg string) {
-	log.Infof("%s: %v %v %v %v (%s)", msg, r.Method, r.URL, r.Proto, r.RemoteAddr,
-		r.Header.Get("X-Forwarded-Proto"))
-	if log.LogVerbose() {
-		for name, headers := range r.Header {
-			for _, h := range headers {
-				log.LogVf("Header %v: %v\n", name, h)
-			}
-		}
-	}
-}
-
 // LogAndAddCacheControl logs the request and wrapps an HTTP handler to add a Cache-Control header for static files.
 func LogAndAddCacheControl(h http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		LogRequest(r, "Static")
+		fhttp.LogRequest(r, "Static")
 		path := r.URL.Path
 		if path == faviconPath {
 			r.URL.Path = "/static/img" + faviconPath // fortio/version expected to be stripped already
@@ -548,7 +520,7 @@ func sendTSVDataIndex(urlPrefix string, w http.ResponseWriter) {
 // LogAndFilterDataRequest logs the data request.
 func LogAndFilterDataRequest(h http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		LogRequest(r, "Data")
+		fhttp.LogRequest(r, "Data")
 		path := r.URL.Path
 		if strings.HasSuffix(path, "/") || strings.HasSuffix(path, "/index.html") {
 			sendHTMLDataIndex(w)
@@ -582,41 +554,6 @@ func LogAndFilterDataRequest(h http.Handler) http.Handler {
 	})
 }
 
-// FetcherHandler is the handler for the fetcher/proxy.
-func FetcherHandler(w http.ResponseWriter, r *http.Request) {
-	LogRequest(r, "Fetch")
-	hj, ok := w.(http.Hijacker)
-	if !ok {
-		log.Critf("hijacking not supported")
-		return
-	}
-	conn, _, err := hj.Hijack()
-	if err != nil {
-		log.Errf("hijacking error %v", err)
-		return
-	}
-	// Don't forget to close the connection:
-	defer conn.Close() // nolint: errcheck
-	url := r.URL.String()[len(fetchPath):]
-	opts := fhttp.NewHTTPOptions("http://" + url)
-	opts.HTTPReqTimeOut = 5 * time.Minute
-	onBehalfOf(opts, r)
-	client := fhttp.NewClient(opts)
-	if client == nil {
-		return // error logged already
-	}
-	_, data, _ := client.Fetch()
-	_, err = conn.Write(data)
-	if err != nil {
-		log.Errf("Error writing fetched data to %v: %v", r.RemoteAddr, err)
-	}
-	client.Close()
-}
-
-func onBehalfOf(o *fhttp.HTTPOptions, r *http.Request) {
-	_ = o.AddAndValidateExtraHeader("X-On-Behalf-Of: " + r.RemoteAddr) // nolint: gas
-}
-
 // TODO: move tsv/xml sync handling to their own file (and possibly package)
 
 // http.ResponseWriter + Flusher emulator - if we refactor the code this should
@@ -658,7 +595,7 @@ func Sync(out io.Writer, u string, datadir string) bool {
 
 // SyncHandler handles syncing/downloading from tsv url.
 func SyncHandler(w http.ResponseWriter, r *http.Request) {
-	LogRequest(r, "Sync")
+	fhttp.LogRequest(r, "Sync")
 	flusher, ok := w.(http.Flusher)
 	if !ok {
 		log.Fatalf("expected http.ResponseWriter to be an http.Flusher")
@@ -677,7 +614,7 @@ func SyncHandler(w http.ResponseWriter, r *http.Request) {
 	w.Write([]byte("Fetch of index/bucket url ... ")) // nolint: gas, errcheck
 	flusher.Flush()
 	o := fhttp.NewHTTPOptions(uStr)
-	onBehalfOf(o, r)
+	fhttp.OnBehalfOf(o, r)
 	// If we had hundreds of thousands of entry we should stream, parallelize (connection pool)
 	// and not do multiple passes over the same data, but for small tsv this is fine.
 	// use std client to change the url and handle https:
@@ -851,23 +788,6 @@ func downloadOne(w http.ResponseWriter, client *fhttp.Client, name string, u str
 	w.Write([]byte("<td class='checkmark'>✓")) // nolint: gas, errcheck
 }
 
-// LogAndCall wrapps an HTTP handler to log the request first.
-func LogAndCall(msg string, hf http.HandlerFunc) http.HandlerFunc {
-	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		LogRequest(r, msg)
-		hf(w, r)
-	})
-}
-
-// SetupPPROF add pprof to the mux (mirror the init() of http pprof).
-func SetupPPROF(mux *http.ServeMux) {
-	mux.HandleFunc("/debug/pprof/", LogAndCall("pprof:index", pprof.Index))
-	mux.HandleFunc("/debug/pprof/cmdline", LogAndCall("pprof:cmdline", pprof.Cmdline))
-	mux.HandleFunc("/debug/pprof/profile", LogAndCall("pprof:profile", pprof.Profile))
-	mux.HandleFunc("/debug/pprof/symbol", LogAndCall("pprof:symbol", pprof.Symbol))
-	mux.HandleFunc("/debug/pprof/trace", LogAndCall("pprof:trace", pprof.Trace))
-}
-
 // Serve starts the fhttp.Serve() plus the UI server on the given port
 // and paths (empty disables the feature). uiPath should end with /
 // (be a 'directory' path)
@@ -878,7 +798,7 @@ func Serve(baseurl, port, debugpath, uipath, staticRsrcDir string, datadir strin
 	if uipath == "" {
 		return
 	}
-	SetupPPROF(mux)
+	fhttp.SetupPPROF(mux)
 	uiPath = uipath
 	dataDir = datadir
 	if uiPath[len(uiPath)-1] != '/' {
@@ -888,7 +808,7 @@ func Serve(baseurl, port, debugpath, uipath, staticRsrcDir string, datadir strin
 	debugPath = ".." + debugpath // TODO: calculate actual path if not same number of directories
 	mux.HandleFunc(uiPath, Handler)
 	fetchPath = uiPath + fetchURI
-	mux.HandleFunc(fetchPath, FetcherHandler)
+	mux.Handle(fetchPath, http.StripPrefix(fetchPath, http.HandlerFunc(fhttp.FetcherHandler)))
 	fhttp.CheckConnectionClosedHeader = true // needed for proxy to avoid errors
 
 	logoPath = version.Short() + "/static/img/logo.svg"
@@ -970,22 +890,6 @@ func Report(baseurl, port, staticRsrcDir string, datadir string) {
 	mux.Handle(uiPath+"data/", LogAndFilterDataRequest(http.StripPrefix(uiPath+"data", fsd)))
 }
 
-// -- Redirection to https feature --
-
-// RedirectToHTTPSHandler handler sends a redirect to same URL with https.
-func RedirectToHTTPSHandler(w http.ResponseWriter, r *http.Request) {
-	dest := "https://" + r.Host + r.URL.String()
-	LogRequest(r, "Redirecting to "+dest)
-	http.Redirect(w, r, dest, http.StatusSeeOther)
-}
-
-// RedirectToHTTPS Sets up a redirector to https on the given port.
-// (Do not create a loop, make sure this is addressed from an ingress)
-func RedirectToHTTPS(port string) {
-	m, _ := fhttp.HTTPServer("https redirector", port)
-	m.HandleFunc("/", RedirectToHTTPSHandler)
-}
-
 // setHostAndPort takes hostport in the form of hostname:port, ip:port or :port,
 // sets the urlHostPort variable.
 func setHostAndPort(inputPort string, addr *net.TCPAddr) {
diff --git a/version/version.go b/version/version.go
index 8dfe7e6..7b38c83 100644
--- a/version/version.go
+++ b/version/version.go
@@ -16,6 +16,7 @@
 package version // import "istio.io/fortio/version"
 import (
 	"fmt"
+	"runtime"
 
 	"istio.io/fortio/log"
 )
@@ -88,5 +89,5 @@ func init() {
 		buildInfo += "-dirty"
 		log.Debugf("gitstatus is %q, marking buildinfo as dirty: %v", gitstatus, buildInfo)
 	}
-	longVersion = version + " " + buildInfo
+	longVersion = version + " " + buildInfo + " " + runtime.Version()
 }


Test patch:
diff --git a/fhttp/http_loglevel_test.go b/fhttp/http_loglevel_test.go
index 306b7d7..93ea44e 100644
--- a/fhttp/http_loglevel_test.go
+++ b/fhttp/http_loglevel_test.go
@@ -29,4 +29,5 @@ func TestDebugMode(t *testing.T) {
 	log.SetLogLevel(log.Debug)
 	TestHTTPRunner(t)
 	TestNoFirstChunkSizeInitially(t)
+	TestFetchAndOnBehalfOf(t)
 }
diff --git a/fhttp/http_test.go b/fhttp/http_test.go
index a8dda29..91ba26a 100644
--- a/fhttp/http_test.go
+++ b/fhttp/http_test.go
@@ -15,11 +15,13 @@
 package fhttp
 
 import (
+	"bytes"
 	"fmt"
 	"io/ioutil"
 	"net"
 	"net/http"
 	"net/url"
+	"os"
 	"strings"
 	"testing"
 	"time"
@@ -725,6 +727,82 @@ func TestDebugHandlerSortedHeaders(t *testing.T) {
 	}
 }
 
+func TestPPROF(t *testing.T) {
+	mux, addr := HTTPServer("test pprof", "0")
+	url := fmt.Sprintf("localhost:%d/debug/pprof/heap?debug=1", addr.Port)
+	code, _ := Fetch(&HTTPOptions{URL: url})
+	if code != http.StatusNotFound {
+		t.Errorf("Got %d instead of expected 404/not found for %s", code, url)
+	}
+	SetupPPROF(mux)
+	code, data := FetchURL(url)
+	if code != http.StatusOK {
+		t.Errorf("Got %d %s instead of ok for %s", code, DebugSummary(data, 256), url)
+	}
+	if !bytes.Contains(data, []byte("TotalAlloc")) {
+		t.Errorf("Result %s doesn't contain expected TotalAlloc", DebugSummary(data, 1024))
+	}
+}
+
+func TestFetchAndOnBehalfOf(t *testing.T) {
+	mux, addr := Serve("0", "/debug")
+	mux.Handle("/fetch/", http.StripPrefix("/fetch/", http.HandlerFunc(FetcherHandler)))
+	url := fmt.Sprintf("localhost:%d/fetch/localhost:%d/debug", addr.Port, addr.Port)
+	code, data := Fetch(&HTTPOptions{URL: url})
+	if code != http.StatusOK {
+		t.Errorf("Got %d %s instead of ok for %s", code, DebugSummary(data, 256), url)
+	}
+	// ideally we'd check more of the header but it can be 127.0.0.1:port or [::1]:port depending on ipv6 support etc...
+	if !bytes.Contains(data, []byte("X-On-Behalf-Of: ")) {
+		t.Errorf("Result %s doesn't contain expected On-Behalf-Of:", DebugSummary(data, 1024))
+	}
+}
+
+func TestRedirector(t *testing.T) {
+	addr := RedirectToHTTPS(":0")
+	relativeURL := "/foo/bar?some=param&anotherone"
+	url := fmt.Sprintf("http://localhost:%d%s", addr.Port, relativeURL)
+	opts := NewHTTPOptions(url)
+	opts.AddAndValidateExtraHeader("Host: foo.istio.io")
+	code, data := Fetch(opts)
+	if code != http.StatusSeeOther {
+		t.Errorf("Got %d %s instead of %d for %s", code, DebugSummary(data, 256), http.StatusSeeOther, url)
+	}
+	if !bytes.Contains(data, []byte("Location: https://foo.istio.io"+relativeURL)) {
+		t.Errorf("Result %s doesn't contain Location: redirect", DebugSummary(data, 1024))
+	}
+}
+
+var testNeedEscape = "<a href='http://google.com'>link</a>"
+
+func escapeTestHandler(w http.ResponseWriter, r *http.Request) {
+	LogRequest(r, "escapeTestHandler")
+	out := NewHTMLEscapeWriter(w)
+	fmt.Fprintln(out, testNeedEscape)
+}
+
+func TestHTMLEscapeWriter(t *testing.T) {
+	mux, addr := HTTPServer("test escape", ":0")
+	mux.HandleFunc("/", escapeTestHandler)
+	url := fmt.Sprintf("http://localhost:%d/", addr.Port)
+	code, data := FetchURL(url)
+	if code != http.StatusOK {
+		t.Errorf("Got %d %s instead of ok for %s", code, DebugSummary(data, 256), url)
+	}
+	if !bytes.Contains(data, []byte("&lt;a href=&#39;http://google.com&#39;&gt;link")) {
+		t.Errorf("Result %s doesn't contain expected escaped html:", DebugSummary(data, 1024))
+	}
+}
+
+func TestNewHTMLEscapeWriterError(t *testing.T) {
+	log.Infof("Expect error complaining about not an http/flusher:")
+	out := NewHTMLEscapeWriter(os.Stdout) // should cause flusher to be null
+	hw := out.(*HTMLEscapeWriter)
+	if hw.Flusher != nil {
+		t.Errorf("Shouldn't have a flusher when not passing in an http: %+v", hw.Flusher)
+	}
+}
+
 // --- for bench mark/comparaison
 
 func asciiFold0(str string) []byte {