ExpandableListConnector: Use children count only for expanded groups

Change-Id: Ib737070ac27d7e2fa18d2f940017f872454a6b05