Issue 1:
I premise I'm opening a ticket only after almost 48 hours of searches & tests.

I tested even your official NFS examples and unofficial docker images and configs - nothing works.

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:15:30Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Google Cloud - Container Engine Cluster (europe-west1-b , 1 node, 1 vCPU, RAM 1.7 GB, node ver. 1.4.0 ) - All permissions on (full/enabled/read-write)

Every command & test is running through Google Cloud Shell VM.

The Problem:

I'm trying to run the same environment (which was working on previous versions of k8s) on Google Container Engine with this config:

1 deployment (1 replica container) as NFS server connected directly to Google Persistent Disk (Standard - 10Gb) - gcr.io/google_containers/volume-nfs image

1 deployment (1 or 2 replicas / containers) as web server - php:7.0.8-apache official docker image

The main scope is to share 1 pd-volume between each webapp-server replica but when I create my deployments, during the web-app deployment creation, it breaks with NFS/mount problem.

(the NFS server starts even if with some warning)

Here some other info about the mount error:

kubectl describe po web-app

....[omitted]...
Mounting arguments: 10.0.2.9:/exports /var/lib/kubelet/pods/...../volumes/kubernetes.io~nfs/pv-nfs-webapp-logs nfs []
Output: mount: wrong fs type, bad option, bad superblock on 10.0.2.9:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)

In some cases useful info is found in syslog - try
dmesg | tail or so.

....[omitted]...

From this moment 10.0.2.9 is the NFS Server host IP

If I connect inside the web-app pod I can ping the NFS server (by IP), and it's sharing the right folders as described:

showmount -e 10.0.2.9

Export list for 10.0.2.9:
/ *
/exports *

The nmap report confirms running ports are OK:

Nmap scan report for 10.0.2.9
Host is up (0.000026s latency).
Not shown: 998 closed ports
PORT STATE SERVICE
111/tcp open rpcbind
2049/tcp open nfs

Here the logs of NFS pod:

kubectl logs nfs-deployment

Serving /exports
Serving /
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused
Starting rpcbind
exportfs: / does not support NFS export
NFS started

kubectl describe po nfs-deployment

Name: nfs-deployment-...
Status: Running
IP: 10.0.2.9
...
Controllers: ReplicaSet/nfs-deployment-115940229
Containers:
nfs-server:
Container ID: docker://.....
Image: gcr.io/google_containers/volume-nfs
Image ID: docker://sha256:...
Ports: 2049/TCP, 111/TCP
Requests:
cpu: 10m
State: Running
Started: Thu, 06 Oct 2016 16:45:40 +0200
Ready: True
Restart Count: 0
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
nfs-pd:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: disk-1
FSType: ext4
Partition: 0
ReadOnly: false
default-token-f7r08:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-f...

I tested many configurations but the same working on older k8s version is like this:

(in order of creation)

nfs.yaml

# NFS POD / SERVER (PERSISTEN DISK SHARING)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nfs-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: nfs-server
role: backend
#env: staging
spec:
restartPolicy: Always
containers:
- name: nfs-server
#image: gcr.io/google-samples/nfs-server:1.1
image: gcr.io/google_containers/volume-nfs
#imagePullPolicy: Always
ports:
#- name: nfs
# containerPort: 2049
# hostPort: 2049
- name: nfs
hostPort: 2049
containerPort: 2049
protocol: TCP
- name: mountd
hostPort: 20048
containerPort: 20048
protocol: UDP
- name: rpcbind
hostPort: 111
containerPort: 111
volumeMounts:
- mountPath: "/exports"
name: nfs-pd
securityContext:
privileged: true
resources:
requests:
cpu: 10m
volumes:
- name: nfs-pd
gcePersistentDisk:
pdName: disk-1
fsType: ext4

pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-nfs-webapp
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
nfs:
#this will be changed after nfs server is ready
server: 10.0.2.7
path: "/exports"
readOnly: false

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-nfs-webapp
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi

web.yaml

# POD DEPLOYMENT
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: webapp-deployment
spec:
#replicas: 1
template:
metadata:
labels:
app: webapp
role: frontend
spec:
restartPolicy: Always
containers:
- name: webapp
image: gcr.io/my-app-reg/my-webapp:v0.0.1
ports:
- name: apache-http
containerPort: 80
- name: apache-https
containerPort: 443
resources:
requests:
cpu: 10m
livenessProbe:
httpGet:
path: /kubernetes-api/status/healthz
port: 80
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 60
timeoutSeconds: 10
readinessProbe:
httpGet:
path: /kubernetes-api/status/readiness
port: 80
scheme: HTTP
initialDelaySeconds: 15
timeoutSeconds: 10
volumeMounts:
- name: nfs-vol-webapp
mountPath: /var/www/html/
volumes:
- name: nfs-vol-webapp
persistentVolumeClaim:
claimName: pvc-nfs-webapp

Issue 2:
Relevant test failures: #33401

I ran the e2e that "create a PV and a pre-bound PVC: test write access" and it fails on GCI.

I debugged a bit and figured that the mount command is failing -

Sep 24 21:17:31.083: INFO: At 2016-09-24 21:11:05 -0700 PDT - event for write-pod-1mzyy: {kubelet e2e-test-vishnuk-minion-group-jo
4u} FailedMount: MountVolume.SetUp failed for volume "kubernetes.io/nfs/13fd01f0-82d6-11e6-959f-42010af00002-nfs-1nmbf" (spec.Name
: "nfs-1nmbf") pod "13fd01f0-82d6-11e6-959f-42010af00002" (UID: "13fd01f0-82d6-11e6-959f-42010af00002") with: mount failed: exit s
tatus 32
Mounting arguments: 10.180.0.10:/exports /var/lib/kubelet/pods/13fd01f0-82d6-11e6-959f-42010af00002/volumes/kubernetes.io~nfs/nfs-
1nmbf nfs []
Output: mount: wrong fs type, bad option, bad superblock on 10.180.0.10:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)

In some cases useful info is found in syslog - try
dmesg | tail or so.

Looking at the kernel logs, I got the following information:

Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered named UNIX socket transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered udp transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered tcp transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: attempt to initialize umh client tracking in a container ignored.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: attempt to initialize legacy client tracking in a container ignored.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: Unable to initialize client recovery tracking! (-22)
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: starting 10-second grace period (net ffff8801e8941840)

I looked at the NFS server logs and found the following:

Serving /exports
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused
Starting rpcbind
exportfs: /exports does not support NFS export
NFS started

I'd like to know if this is a test setup issue or a base image issue? NFS has been confirmed to be working on GCI by other users of GCI. Even then, I'd like to understand the reason for the NFS PV test failure in GCI.

cc @matchstick @saad-ali @kubernetes/sig-storage @thockin @Amey-D