Issue 1:
I'm very sorry about the quality of this issue report, I've yet to be able to resolve it to a smaller reproduction.

The background is we have an API server which instruments types and exposes their methods over an RPC interface. This stack trace is from the sever side, attempting to dispatch to agent.(*AgentAPIV1).StateServingInfo

unexpected fault address 0xf8010260f8010260
fatal error: fault
[signal 0xb code=0x1 addr=0xf8010260f8010260 pc=0x1e56c]

goroutine 229 [running]:
runtime.throw(0xf45d00, 0x5)
/home/ubuntu/go/src/runtime/panic.go:543 +0x8c fp=0xc2083ff688 sp=0xc2083ff670
runtime.sigpanic()
/home/ubuntu/go/src/runtime/sigpanic_unix.go:27 +0x2f0 fp=0xc2083ff6d8 sp=0xc2083ff688
runtime.convI2E(0xf8010260f8010258, 0x7c030378e8410070, 0x0, 0x0)
/home/ubuntu/go/src/runtime/iface.go:256 +0x74 fp=0xc2083ff708 sp=0xc2083ff6e0
github.com/juju/juju/apiserver/agent.(*AgentAPIV0).StateServingInfo(0x31f5c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93 +0xa0 fp=0xc2083ff830 sp=0xc2083ff708
github.com/juju/juju/apiserver/agent.(*AgentAPIV1).StateServingInfo(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
<autogenerated>:22 +0x40 fp=0xc2083ff838 sp=0xc2083ff830
runtime.call128(0xc208205bd0, 0xe85098, 0xc208398380)
/home/ubuntu/go/src/runtime/asm_ppc64x.s:413 +0x88 fp=0xc2083ff8c0 sp=0xc2083ff838
reflect.Value.call(0xe84f40, 0xc20802c5f0, 0x913, 0xf30060, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/go/src/reflect/value.go:432 +0xe30 fp=0xc2083ffc00 sp=0xc2083ff8c0
reflect.Value.Call(0xe84f40, 0xc20802c5f0, 0x913, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/ubuntu/go/src/reflect/value.go:300 +0x94 fp=0xc2083ffc60 sp=0xc2083ffc00
github.com/juju/juju/rpc/rpcreflect.newMethod.func6(0xe84f40, 0xc20802c5f0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/src/github.com/juju/juju/rpc/rpcreflect/type.go:326 +0x1a0 fp=0xc2083ffd60 sp=0xc2083ffc60
github.com/juju/juju/apiserver.(*srvCaller).Call(0xc208581500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/src/github.com/juju/juju/apiserver/root.go:131 +0xf0 fp=0xc2083ffde8 sp=0xc2083ffd60
github.com/juju/juju/rpc.(*Conn).runRequest(0xc208472be0, 0x3fff7fc0ee38, 0xc208581500, 0x1286728, 0x3, 0xc2080a8460, 0x5, 0x1, 0x0, 0x0, ...)
/home/ubuntu/src/github.com/juju/juju/rpc/server.go:552 +0x114 fp=0xc2083fff28 sp=0xc2083ffde8
runtime.goexit()
/home/ubuntu/go/src/runtime/asm_ppc64x.s:1132 +0x4 fp=0xc2083fff28 sp=0xc2083fff28
created by github.com/juju/juju/rpc.(*Conn).handleRequest
/home/ubuntu/src/github.com/juju/juju/rpc/server.go:481 +0x5f4

however AgentAPIV1 is defined as

// AgentAPIV1 implements the version 1 of the API provided to an agent.
type AgentAPIV1 struct {
*AgentAPIV0
}

So the compiler has autogenerated AgentAPIV1.StateServingInfo and forwarded it to AgentAPIV0.StateServingInfo

github.com/juju/juju/apiserver/agent.(*AgentAPIV0).StateServingInfo(0x31f5c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93 +0xa0 fp=0xc2083ff830 sp=0xc2083ff708
github.com/juju/juju/apiserver/agent.(*AgentAPIV1).StateServingInfo(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
<autogenerated>:22 +0x40 fp=0xc2083ff838 sp=0xc2083ff830

However, looking at the receiver value for agent.(*AgentAPIV1).StateServingInfo it is 0x00, but in the next frame the receiver is 0x31f5c8, this may be related to the problem.

The faulting line, /home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93, is

func (api *AgentAPIV0) StateServingInfo() (result state.StateServingInfo, err error) {
auth := api.auth
fmt.Printf("%#v\n", auth) // line 93

and at this point the value of auth is garbage, its interface type and value are random junk, runtime.convI2E(0xf8010260f8010258, 0x7c030378e8410070, 0x0, 0x0)

So far I have been able to determine that manually adding the forwarding method

func (api *AgentAPIV1) StateServingInfo() (result state.StateServingInfo, err error) {
return api.AgentAPIV0.StateServingInfo()
}

Stops the panic, but I have not yet been able to construct a stand alone reproduction.

Sadly this only happens on our ppc64le system, which is not accessible to others.

/cc @randall77 @minux @aclements

Issue 2:
What steps will reproduce the problem?

I'm starting a gotest of some code that uses lots of goroutines, but the process very
early (before even running any useful code).

I find this issue reproduces more readily on a specific machine x86_64, 8 cores, Fedora
14, 2.6.35.11-83.fc14.x86_64 and if I set GOMAXPROCS=500 and ulimit -u 256 although
neither of these should have any impact on anything.

What do you see instead?

Process hangs until I terminate it with SIGQUIT.

Which compiler are you using (5g, 6g, 8g, gccgo)?

6g

Which operating system are you using?

linux

Which revision are you using? (hg identify)

tip

Please provide any additional information below.

I've attached the full stacktrace and strace output from a run that hangs.

SIGQUIT: quit
PC=0x414b15

runtime.futex+0x23 /home/alberts/go.hg/src/pkg/runtime/linux/amd64/sys.s:147
runtime.futex()
futexsleep+0x50 /home/alberts/go.hg/src/pkg/runtime/linux/thread.c:52
futexsleep(0x6698fc, 0x300000003, 0x414e85, 0x40ed9f)
futexlock+0x85 /home/alberts/go.hg/src/pkg/runtime/linux/thread.c:120
futexlock(0x6698fc, 0x0)
runtime.notesleep+0x25 /home/alberts/go.hg/src/pkg/runtime/linux/thread.c:205
runtime.notesleep(0x6698fc, 0x4da724fd)
runtime.stoptheworld+0x68 /home/alberts/go.hg/src/pkg/runtime/proc.c:426
runtime.stoptheworld()
runtime.gc+0x15c /home/alberts/go.hg/src/pkg/runtime/mgc0.c:561
runtime.gc(0xf800000000, 0x8)
runtime.mallocgc+0x1c3 /home/alberts/go.hg/src/pkg/runtime/malloc.c:90
runtime.mallocgc(0x8, 0x100000000, 0x1, 0x267f900, 0x409a20, ...)

Attachments:

strace.txt (28310 bytes)

stacktrace.txt (2804 bytes)