Repository: deis/deis

Issue:
underscores in app names cause logger to fail
I setup a new cluster from master. I tried stop and start but I still see the logger going into the "failed" state. 

$ fleetctl journal deis-logger

-- Logs begin at Wed 2014-09-17 15:16:23 UTC, end at Wed 2014-09-17 21:29:41 UTC. --
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/net/udpsock_posix.go:82 +0x142
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: github.com/deis/deis/logger/syslog.(_Server).receiver(0xc2080182d0, 0x7f5cd6d384f8, 0xc20803a028)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /go/src/github.com/deis/deis/logger/syslog/server.go:106 +0xab
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: created by github.com/deis/deis/logger/syslog.(_Server).Listen
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /go/src/github.com/deis/deis/logger/syslog/server.go:67 +0x286
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal systemd[1]: deis-logger.service: main process exited, code=exited, status=1/FAILURE
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal systemd[1]: deis-logger.service: control process exited, code=exited status=1
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal systemd[1]: Unit deis-logger.service entered failed state.
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal docker[13190]: Error response from daemon: No such container: deis-logger
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal docker[13190]: 2014/09/17 19:58:40 Error: failed to stop one or more containers



Discussion:
@cmshetty could you please try `fleetctl ssh deis-logger journalctl --no-pager -u deis-logger` and make sure that you're on master? I know that we had to fix the logger in a recent commit, and the start of the log looks like a panic dump to me. The command I listed should dump the whole logs.

I checked I am on master. It is a panic dump - could not find app name in message.

Sep 17 19:35:51 ip-10-21-1-61.ec2.internal systemd[1]: Starting deis-logger...
Sep 17 19:35:51 ip-10-21-1-61.ec2.internal systemd[1]: Started deis-logger.
Sep 17 19:35:51 ip-10-21-1-61.ec2.internal sh[6305]: deis-logger running...
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: 2014-09-17 19:36:20 staging[None.3]: Worker gevent_workers.txt:bfad36e4-3e9e-11e4-9282-4ed38402919f: [cloudapi] [17/Sep/2014 14:36:20: DEBUG/MainProcess:Dummy-1] [gevent_worker:cleanup_workers:90] Number of Workers 1, My Index 0.
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: panic: Could not find app name in message
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: goroutine 21 [running]:
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: runtime.panic(0x54bfe0, 0xc2080003d0)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: main.(_handler).mainLoop(0xc20803a020)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /go/src/github.com/deis/deis/logger/syslogd/syslogd.go:90 +0x21b
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: created by main.newHandler
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /go/src/github.com/deis/deis/logger/syslogd/syslogd.go:31 +0xe1
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: goroutine 16 [chan receive]:
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: main.main()
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /go/src/github.com/deis/deis/logger/syslogd/syslogd.go:106 +0x296
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: goroutine 19 [finalizer wait]:
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: runtime.park(0x414370, 0x660278, 0x65ede9)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: runtime.parkunlock(0x660278, 0x65ede9)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: runfinq()
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: runtime.goexit()
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/runtime/proc.c:1445
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: goroutine 20 [syscall]:
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: os/signal.loop()
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: created by os/signal.init·1
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x32
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: goroutine 22 [IO wait]:
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: net.runtime_pollWait(0x7fedef8894c0, 0x72, 0x0)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: net.(_pollDesc).Wait(0xc2080280d0, 0x72, 0x0, 0x0)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: net.(_pollDesc).WaitRead(0xc2080280d0, 0x0, 0x0)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: net.(_netFD).readFrom(0xc208028070, 0xc208062000, 0x400, 0x400, 0x0, 0x0, 0x0, 0x7fedef8882b8, 0xb)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/net/fd_unix.go:269 +0x3db
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: net.(_UDPConn).ReadFromUDP(0xc20803a028, 0xc208062000, 0x400, 0x400, 0xc208040040, 0x0, 0x0, 0x0)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/net/udpsock_posix.go:67 +0x129
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: net.(_UDPConn).ReadFrom(0xc20803a028, 0xc208062000, 0x400, 0x400, 0xb9, 0x0, 0x0, 0x0, 0x0)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /usr/local/go/src/pkg/net/udpsock_posix.go:82 +0x142
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: github.com/deis/deis/logger/syslog.(_Server).receiver(0xc2080182d0, 0x7fedef8884f8, 0xc20803a028)
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /go/src/github.com/deis/deis/logger/syslog/server.go:106 +0xab
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: created by github.com/deis/deis/logger/syslog.(_Server).Listen
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: /go/src/github.com/deis/deis/logger/syslog/server.go:67 +0x286
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal systemd[1]: deis-logger.service: main process exited, code=exited, status=1/FAILURE
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal docker[6864]: Error response from daemon: No such container: deis-logger
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal docker[6864]: 2014/09/17 19:36:20 Error: failed to stop one or more containers
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal systemd[1]: deis-logger.service: control process exited, code=exited status=1
Sep 17 19:36:20 ip-10-21-1-61.ec2.internal systemd[1]: Unit deis-logger.service entered failed state.
Sep 17 19:43:54 ip-10-21-1-61.ec2.internal systemd[1]: Stopped deis-logger.
Sep 17 19:46:43 ip-10-21-1-61.ec2.internal systemd[1]: Starting deis-logger...
Sep 17 19:46:43 ip-10-21-1-61.ec2.internal systemd[1]: Started deis-logger.
Sep 17 19:46:44 ip-10-21-1-61.ec2.internal sh[7485]: deis-logger running...
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: 2014-09-17 19:58:35 staging[None.3]: Worker gevent_workers.txt:01b6e002-3ea5-11e4-8e0b-4a7d2c802327: [cloudapi] [17/Sep/2014 14:58:35: DEBUG/MainProcess:Dummy-1] [gevent_worker:setup_worker:67] Setup Done Waiting 15 seconds.
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: panic: Could not find app name in message
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: goroutine 21 [running]:
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: runtime.panic(0x54bfe0, 0xc2080003d0)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: main.(_handler).mainLoop(0xc20803a020)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /go/src/github.com/deis/deis/logger/syslogd/syslogd.go:90 +0x21b
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: created by main.newHandler
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /go/src/github.com/deis/deis/logger/syslogd/syslogd.go:31 +0xe1
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: goroutine 16 [chan receive, 9 minutes]:
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: main.main()
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /go/src/github.com/deis/deis/logger/syslogd/syslogd.go:106 +0x296
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: goroutine 19 [finalizer wait, 11 minutes]:
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: runtime.park(0x414370, 0x660278, 0x65ede9)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: runtime.parkunlock(0x660278, 0x65ede9)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: runfinq()
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: runtime.goexit()
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/runtime/proc.c:1445
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: goroutine 20 [syscall, 11 minutes]:
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: os/signal.loop()
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: created by os/signal.init·1
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x32
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: goroutine 22 [IO wait]:
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: net.runtime_pollWait(0x7f5cd6d394c0, 0x72, 0x0)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: net.(_pollDesc).Wait(0xc2080280d0, 0x72, 0x0, 0x0)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: net.(_pollDesc).WaitRead(0xc2080280d0, 0x0, 0x0)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: net.(_netFD).readFrom(0xc208028070, 0xc20806a000, 0x400, 0x400, 0x0, 0x0, 0x0, 0x7f5cd6d382b8, 0xb)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/net/fd_unix.go:269 +0x3db
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: net.(_UDPConn).ReadFromUDP(0xc20803a028, 0xc20806a000, 0x400, 0x400, 0xc208040020, 0x0, 0x0, 0x0)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/net/udpsock_posix.go:67 +0x129
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: net.(_UDPConn).ReadFrom(0xc20803a028, 0xc20806a000, 0x400, 0x400, 0xb4, 0x0, 0x0, 0x0, 0x0)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /usr/local/go/src/pkg/net/udpsock_posix.go:82 +0x142
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: github.com/deis/deis/logger/syslog.(_Server).receiver(0xc2080182d0, 0x7f5cd6d384f8, 0xc20803a028)
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /go/src/github.com/deis/deis/logger/syslog/server.go:106 +0xab
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: created by github.com/deis/deis/logger/syslog.(_Server).Listen
Sep 17 19:58:35 ip-10-21-1-61.ec2.internal sh[7485]: /go/src/github.com/deis/deis/logger/syslog/server.go:67 +0x286
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal systemd[1]: deis-logger.service: main process exited, code=exited, status=1/FAILURE
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal systemd[1]: deis-logger.service: control process exited, code=exited status=1
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal systemd[1]: Unit deis-logger.service entered failed state.
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal docker[13190]: Error response from daemon: No such container: deis-logger
Sep 17 19:58:40 ip-10-21-1-61.ec2.internal docker[13190]: 2014/09/17 19:58:40 Error: failed to stop one or more containers

> Sep 17 19:36:20 ip-10-21-1-61.ec2.internal sh[6305]: 2014-09-17 19:36:20 staging[None.3]: Worker gevent_workers.txt:bfad36e4-3e9e-11e4-9282-4ed38402919f: [cloudapi] [17/Sep/2014 14:36:20:

Looks like it's failing on the process type's name. The regex in question is here: https://github.com/deis/deis/blob/master/logger/syslogd/syslogd.go#L48

`None.3` contains a capital letter, which is an invalid character. We should allow capital letters for process types.

The  process type shouldn't be None. 

$ deis ps
=== staging Processes

--- web: 
web.1 up (v3)

--- all_worker: 
all_worker.1 up (v3)

--- gevent_worker: 
gevent_worker.1 up (v3)

something tells me that the underscore in the process type is causing shenanigans in the platform. can you check `deis info` for me please? Also, are you able to reproduce this issue consistently?

Yes I have restarted the deis-logger and it keeps going into panic.

Here is the deis info:

=== staging Application
{
  "updated": "2014-09-17T19:58:49.097Z", 
  "uuid": "d990d6fe-fce5-4d5f-829d-3baac67a40df", 
  "created": "2014-09-17T18:52:56.709Z", 
  "url": "staging.sookasa.com", 
  "cluster": "dev", 
  "owner": "dev", 
  "id": "staging", 
  "structure": "{\"web\": 1, \"all_worker\": 1, \"gevent_worker\": 1}"
}

=== staging Processes

--- web: 
web.1 up (v3)

--- all_worker: 
all_worker.1 up (v3)

--- gevent_worker: 
gevent_worker.1 up (v3)

=== staging Domains
No domains


Resolving patch:
diff --git a/controller/scheduler/coreos.py b/controller/scheduler/coreos.py
--- a/controller/scheduler/coreos.py
+++ b/controller/scheduler/coreos.py
@@ -11,7 +11,7 @@
 
 
 MATCH = re.compile(
-    '(?P<app>[a-z0-9-]+)_?(?P<version>v[0-9]+)?\.?(?P<c_type>[a-z]+)?.(?P<c_num>[0-9]+)')
+    '(?P<app>[a-z0-9-]+)_?(?P<version>v[0-9]+)?\.?(?P<c_type>[a-z-_]+)?.(?P<c_num>[0-9]+)')
 
 
 class UHTTPConnection(httplib.HTTPConnection):


Test patch: