socket/glusterd/client: enable SSL for management

The feature is controlled by presence of the following file:

	/var/lib/glusterd/secure-access

See the comment near the definition of SECURE_ACCESS_FILE in glusterfs.h
for the rationale.  With this enabled, the following rules apply to
connections:

	UNIX-domain sockets never have SSL.

	Management-port sockets (both connecting and accepting, in
	daemons and CLI) have SSL based on presence of the file.

	Other IP sockets have SSL based on the existing client.ssl and
	server.ssl volume options.

Transport multi-threading is explicitly turned off in glusterd (it would
otherwise be turned on when SSL is) due to multi-threading issues.
Tests have been elided to avoid risk of leaving a file which will cause
all subsequent tests to run with management SSL still enabled.

IMPLEMENTATION NOTE
The implementation is a bit messy, and consists of two stages.  First we
decide whether to set the relevant fields in our context structure, based
on presence of the sentinel file OR a command-line override.  Later we
decide whether a particular connection should actually use SSL, based on the
context flags plus what kind of connection we're making[1] and what kind of
daemon we're in[2].

[1] inbound, outbound to glusterd port, other outbound
[2] glusterd, glusterfsd, other

TESTING NOTE
Instead of just running one special test for this feature, the ideal
would be to run all tests with management SSL enabled.  However, it
would be inappropriate or premature to set up an optional feature in the
patch itself.  Therefore, the method of choice is to submit a separate
patch on top, which modifies "cleanup" in include.rc to recreate the
secure-access file and associated SSL certificate/key files before each
test.

Change-Id: I0e04d6d08163893e24ec8c031748c5c447d7f780
BUG: 1114604
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.org/8094
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterd/regression: Temp fix for spurious err

As discussed in the mail,
Disabling the checking of snap brick status until
the investigation is done on the port bind issue.

Change-Id: I8854cee050de1b7f843e3d40631b6cb61fd8583e
BUG: 1112559
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/8259
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
features/changelog: prevent deadlock on thread cancellation

helper threads (fsync, rollover) wake up periodically and perform
their respective operation under a lock (crt->lock). These threads
are also subjected to cancellation under some circumstance such as
disabling changelog. This is inherently dangerous when funtions
which are cancellation points for pthread_cancel(3) are used
in the locked region.

Consider this

         pthread_mutex_lock(&mutex);
         {
                /* ... */
                ret = fsync (fd);   <-- cancellation point
                /* ... */
         }
         pthread_mutex_unlock(&mutex);

A pthread_cancel(3) by another thread just before fsync(3) but
after pthread_mutex_lock(3) would result in the thread getting
cancelled when fsync(3) is invoked, thereby never unlocking the
mutex. Moreover, in case of changelog translator, the locked
region (under crt->lock in changelog-rt.c) is also the code
path for fop changelog updation. Therefore, unlocking the
mutex in thread cleanup handler (pthread_cleanup_pop(3)) might
prematurely release the mutex during fop updation path.

This patch fixes such problems existing in fsync and rollover
threads. Fix is to enter the locked region with cancellation
disabled and enable it after mutex unlock. Also, test for a
cancellation request early on in case none of the functions
are cancellation points.

Change-Id: I1795627a12827609c1da659d07fc1457ffa033de
BUG: 1110917
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Reviewed-on: http://review.gluster.org/8106
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
tests: Fixed spurious failure in bug-887098-gmount-crash.t

This script was trying to see if the mount process died by doing a
'ps ax' and a grep of the original pid in the results. After that
the pid of the first line returned by grep was compared to the
original pid.

This method can lead to false negatives because it's possible that
the original pid appears in some other part of the 'ps ax' list.

This patch uses get_mount_process_pid() from volume.rc to check if
the process is still alive.

Change-Id: I0285366e601a146793c47e9c1156a4bb36d6fcb3
BUG: 1092850
Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
Reviewed-on: http://review.gluster.org/8286
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/ec: Added erasure code translator

Change-Id: I293917501d5c2ca4cdc6303df30cf0b568cea361
BUG: 1118629
Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
Reviewed-on: http://review.gluster.org/7749
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cli/glusterd: Added support for dispersed volumes

Two new options have been added to the 'create' command of the cli
interface:

    disperse [<count>] redundancy <count>

Both are optional. A dispersed volume is created by specifying, at
least, one of them. If 'disperse' is missing or it's present but
'<count>' does not, the number of bricks enumerated in the command
line is taken as the disperse count.

If 'redundancy' is missing, the lowest optimal value is assumed. A
configuration is considered optimal (for most workloads) when the
disperse count - redundancy count is a power of 2. If the resulting
redundancy is 1, the volume is created normally, but if it's greater
than 1, a warning is shown to the user and he/she must answer yes/no
to continue volume creation. If there isn't any optimal value for
the given number of bricks, a warning is also shown and, if the user
accepts, a redundancy of 1 is used.

If 'redundancy' is specified and the resulting volume is not optimal,
another warning is shown to the user.

A distributed-disperse volume can be created using a number of bricks
multiple of the disperse count.

Change-Id: Iab93efbe78e905cdb91f54f3741599f7ea6645e4
BUG: 1118629
Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
Reviewed-on: http://review.gluster.org/7782
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
DHT/Logging
Changed the log level of a message from none to debug as none does
not print a log level in the log file.

Change-Id: I463d1095d69bbd0036958282da13cb8e0226f34f
BUG: 1116797
Signed-off-by: Nithya Balachandran <nbalacha@redhat.com>
Reviewed-on: http://review.gluster.org/8253
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
dht: support heterogeneous brick sizes

Calculation of layouts now considers the size of each brick, so that
smaller bricks don't get an "unfair" share of allocations and start
returning ENOSPC while the larger bricks still have plenty of space.

The observation has been made that some clients might get ENOTCONN when
trying to fetch disk-size information, and end up calculating layouts
differently.  The following meta-observations can be made.

(1) This scenario is extremely unlikely in configurations with AFR.

(2) The most likely consequence of this scenario is that some files will
be placed sub-optimally by the client with the obsolete (non-weighted)
layout.  They'll still be found anyway, so this isn't a show stopper.

(3) Without this patch it's *guaranteed* that some files will be placed
sub-optimally, because any layout that fails to account for brick sizes
is sub-optimal.

(4) We shouldn't be doing fix-layout from two nodes simultaneously
anyway.  That's inefficient at best.  Any instances of such behavior are
separate bugs, which should be fixed separately.

(5) In the most extreme edge case, two nodes doing weighted and
non-weighted layout fixes could race and end up creating an internally
inconsistent layout.  This condition is still transient; it will be
detected and repaired automatically the next time anyone fetches the
layout.  (If it's not that's also a preexisting bug that can show up in
other contexts.)

In conclusion, it's not the purpose of this patch to fix bugs elsewhere
in DHT.  Its purpose is to make life incrementally better for users who
add new hardware with larger disks etc. than the older equipment.  It's
only one part of an ongoing process to improve layout management and
repair, all the way up to support for multiple hash rings or tiering.

Change-Id: I05eb6f9eface9cdaf8622e0260c8c7f29020447f
BUG: 1114680
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.org/8093
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cli/snapshot: provide --xml support for all snapshot command

Now --xml option can be used with all snapshot command. It
will form the cli output in xml form.

Change-Id: Ifc0ac31d2a9f91e136e87f3b51a629df7dba94e8
BUG: 1096610
Signed-off-by: Rajesh Joseph <rjoseph@redhat.com>
Reviewed-on: http://review.gluster.org/7663
Reviewed-by: Sachin Pandit <spandit@redhat.com>
Reviewed-by: Vijaikumar Mallikarjuna <vmallika@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterd/snapshot: Update fstype for local bricks only

While creating snapshot, update fstype for local bricks only
and not for bricks hosted on other nodes

Also returning ret as 0, in case no cleanup is required in
post-validation, so that a post-validation failure is not
logged, every time a pre-validation failure happens.

Change-Id: I6364e33cfd9528e0a988ee48f3443239ee884336
BUG: 1111060
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/8272
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Tested-by: Krishnan Parthasarathi <kparthas@redhat.com>
features/gfid-access: error handling for entry creation

Proceed with setattr() only on a successfull entry creation.
Winding a setattr() using a freshlyOC initiated inode would
most likely fail in one translator or the other (e.g. DHT
expecting the layout information to be set in the inode
context), which is the case if the inode was not looked up.

Therefore, gfid-access handles failure entry creations and
passes the _correct_ errno back to the client instead of
continuing with setattr() call and probably returning back
incorrect errno. Also, filling up inode->gfid is required
as the new inode is not looked up and ->gfid would be
certainely required for inode operations.

Change-Id: Ie92f5647a89bf558c07710ab0400bce69d59fc31
BUG: 1111490
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Reviewed-on: http://review.gluster.org/8260
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
protocol/server: Add extra space between words

Change-Id: Id658db992a43202602e0c30d2c8ca639c4aeadd6
BUG: 1119167
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/8304
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Regression test portability: dd usage

NetBSD, FreeBSD, and MacOS X dd(1) bs argument uses m for megabyte, while
Linux uses M. Use bs=1024k instead of bs=1M for better compatibility.

BUG: 764655
Change-Id: I603f57adbc9b31f6d634b918726437fbfce42e03
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Reviewed-on: http://review.gluster.org/8278
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-by: Justin Clift <justin@gluster.org>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
test: utility to create files with a specific gfid

python test utility that uses setxattr() interface to
create files with a predefined gfid (which is also used
by geo-replication). this utility aids in writing test
cases for gfid-access translator.

Change-Id: Ib11248844a094cc02dae74d969f2e9212f94ede4
BUG: 1111490
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Reviewed-on: http://review.gluster.org/8275
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
changelog: variable 'writen' is changed

This patch change the variable name to 'written' in changelog, along
with few other typo fixing.

Change-Id: I1d3cbea5c89fc0a15233bb6e61b4fbd4bfdd3a67
BUG: 1075417
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
Reviewed-on: http://review.gluster.org/8267
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Tested-by: Venky Shankar <vshankar@redhat.com>
Autoload fuse module in centos5/el5 versions.

This patch dynamically load 'fuse' module with the installation
of glusterfs-fuse package in centos/el5 distributions.

Change-Id: I4988a025bd18716fd6d355431196540e140224b3
BUG: 1108669
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
Reviewed-on: http://review.gluster.org/8116
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Reviewed-by: Lalatendu Mohanty <lmohanty@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: Fix dirent leak

afr does itransform by taking the list of entries given by client xlator
to separate list but doesn't free that list.

Change-Id: Ibb4d38b4934b2bb924385c88f9d7942fad933cb9
BUG: 1117243
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/8261
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-by: Ravishankar N <ravishankar@redhat.com>
gfapi : remove gfapi.py to avoid confusion

python binding of libgfapi is evolving via libgfapi-python
project. The example available here confuses end users,
so removing it from this path.

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
Bug Id: 1119328
> Reviewed-on: http://review.gluster.org/7920
> Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
> Tested-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
> Reviewed-by: Thiago da Silva <thiago@redhat.com>
> Reviewed-by: Prashanth Pai <ppai@redhat.com>
> Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Change-Id: I17b3aa0f0505342496019ce012cca21d84184027
Reviewed-on: http://review.gluster.org/8237
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterd: Fix for resource leak coverity bug 1223045.

Change-Id: I96261e7f5cd7b5550d3100750c80190dd932a8ab
BUG: 789278
Signed-off-by: Raghavendra Talur <rtalur@redhat.com>
Reviewed-on: http://review.gluster.org/8252
Reviewed-by: Avra Sengupta <asengupt@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
cli: Changed "rebalance start" output

Change-Id: Ie87f1a2107b07a6e519ed894a74edf3b3e0a8340
BUG: 1063230
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/6946
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>