This is the freeware version of sha-1 provided by Paul E. Jones (see
license.txt), with minor modifications to fit into iRODS.  Where I've
modified a .h or .c file, I've kept the original in a _.h.orig or
_.c.orig file.  The sha.c and shatest.c are test programs in the
original that can be linked in with it.  sha1.c contains the sha
library functions used in iRODS.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5412 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Optionally use SHA-1 instead of MD5 in iRODS password authentication
and obfuscation.

This is for sites that have a security policy requiring software to
not use MD5 due to weaknesses discovered in it.  However, the
weaknesses found do not actually affect iRODS authentication as the
primary property needed for iRODS is simply the one-way aspect of the
hash (i.e. one can't derive the password from its hash (so that
monitoring the network traffic doesn't reveal the password)), and MD5
is quite strong in this regard.  The MD5 flaw is that it is not as
collision resistant as previously thought; it is possible for one to
design a file that has different content but the same MD5 hash as
another file.

In the iRODS authentication protocol a challenge value is hashed with
the password, which is returned to the server as proof that the
password is known.  As long as this hash is one-way and collisions are
rare, this is fine.  Even with the MD5 limitations, collisions are
extremely rare.  And since the challenge is unique (64 pseudo-random
bytes), one cannot attack the authentication sub-system with prepared
responses.

SHA-1 has since been found to be vulnerable in this manner as well,
altho it, like MD5, is still completely sufficient for iRODS
authentication.

----

Currently, to enable SHA-1 as the default authentication/obfuscation
hash edit the lib/core/src/obf.c and change the line:
static int defaultHashType=HASH_TYPE_MD5;
to:
static int defaultHashType=HASH_TYPE_SHA1;

I may add a micro-service/rule so that this can be set by updating a
rule but, for now, you can edit this setting.  There's also a
function, obfSetDefaultHashType, that can used to set this (this is
used in iadmin for some tests).

To maintain good backward compatability, and well as between the
various clients and servers and for inter-zone operations, the iRODS
system and protocol will determine which hash is available and fall
back to MD5 when needed.  For example, for a site that configures
SHA-1 as preferred, the servers will indicate to clients that SHA-1 is
preferred and if the client is SHA-1 enabled, it will recognize this
and return to the server a SHA-1 hash.  The server is able to tell if
the client actually used SHA-1 in the returned values.  This
complicates the implementation (actually represents most of the
development effort), but is critical for this SHA-1 capability to be
available and reliable.

For iRODS obfuscation capabilities, many various cases also now use
SHA-1 instead for MD5 when SHA-1 is preferred.  As for authetication,
the system is able to determine if SHA-1 was used and switch back to
MD5 if not.  This is only obfucation though, and with the iRODS source
code one can de-scramble obfuscated values.  It does avoid completely
plain text tho, and avoids inadvertant disclosure of values.  But it
is not critical that SHA1 is used in all cases, as both are fine.

The obfEncodeByKey function will now add the string 'sha1' at the
beginning of the output string and the obfDecodeByKey function will
look for this at the beginning of the input string and if found, will
skip past it use sha1 to decode.

A open-source version of sha-1 has been added to the iRODS source tree
under lib/sha1 and is now built with iRODS in a manner similar to how
MD5 is.  See lib/sha1/README.txt for more.

There is a new iadmin sub-commands to test SHA-1: 'sethash', 'md5',
and 'sha1'.  See the diffs and code for how these work. I didn't add
them to the help text since they are just for test.

I also plan to modify the DICE continuous build/test system to
configure an instance with SHA-1 enabled occasional (probably daily).
Since there are no new ICAT SQL forms with this, no new SQL-specific
tests were needed.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5413 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Add a daily test for tinderbox (continuous build/test) with SHA1 as
the default hash.  


git-svn-id: svn://irodssvn.ucsd.edu/trunk@5414 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Improve an error message and the error return code for an 'ichmod
inherit' case.  The error return is now CAT_NO_ACCESS_PERMISSION,
which is likely (but not certain). This problem was noticed by
Jean-Yves.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5415 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Add option "-s" to iphybun: maximum size for a tar bundle file (default is 4).
Implement this option and the newly created "-R" option to the corresponding msi: msiPhyBundleColl.


git-svn-id: svn://irodssvn.ucsd.edu/trunk@5416 ea137bd6-aa86-47b6-b90d-b101d9f19a90
The following changes allow iRODS to be built with Boost enabled again
on Linux systems using the system-installed Boost instead of the
version included in the iRODS release/repository.  Next steps will be
to remove the boost library source code and support from the
repository.

Previously, in collaboration with RENCI, supoort for Boost was added
in iRODS 3.0, primarily to make the Windows version immediately
available.  For Linux/Unix, Boost was a build-time option but has not
been working recently (3.2 and probably 3.1).  The DICE continuous
test system (tinderbox) builds with Boost enabled daily, but the test
was inadvertently not failing when the (re) make failed (this was
fixed a few weeks ago).  Windows support also became a lower priority
effort (altho DICE and RENCI do still support it, and this commit is
in support of that effort).

Additional changes made in the iRODS code post-3.0 make the current
iRODS code incompatible with the version of Boost in the iRODS
repository.  But boost can now be easily installed as a system library
using standard package managers and the version available with recent
Linux releases is sufficient.  This is being tested under Ubuntu
12.04.2 which includes Boost 1.46.1.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5417 ea137bd6-aa86-47b6-b90d-b101d9f19a90
ICAT performance improvements provided by Thomas Ledoux, adding 
'limit 1' when one row is expected (Postgres or MySQL).

These changes add a 'limit 1' string to SQL being run from ICAT
functions in which only one row should be returned which, in some
cases, can significantly improve performance.  Thomas found this
analysing a findAVU call which took several minutes on his large
Postgres instance but with this change took only 36ms! In addition to
that, this change applies to other cases where one row is expected so
there are likely other performance improvements as part of this.

I believe this is safe as the ICAT code is expecting one row and if
something is wrong and multiple rows are returned, it will discard
them anyway.  So the 'limit 1' should do no harm.  In the local
testing I've done (mostly 'devtest') everything still seems to work
fine.

In general, one wouldn't add 'limit 1' in a case like this because it
may be a bit unclear (a reader might wonder why the SQL writer needed
to limit the results to only 1).  But apparently, the DMBS (at least
Postgres) uses this to return once a result is found (reasonably
enough) and so can be much faster.  In local testing on a very small
instance, there is no noticable slow-down using 'limit 1' so it is
likely preferrable in all cases.

Oracle does not support 'limit' but uses a 'where rownum <' construct
so it would need to be done differently and I'm not sure it would be
faster (probably is though).  So this update does not change Oracle
SQL.

This has been tested with Postgres but has not yet been tested with
MySQ (it will later tho and should be fine).

Thanks to Thomas Ledoux of the French National Library for this.
(These are largely Thomas' updates with a couple minor changes made by
me.)

See the irod-chat thread 'Heavy selects for findAVU' for more.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5418 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Remove boost compressed source file and build script, make an
additional update to a test script to remove a now unneeded setting.

This is a continuation of r5417 making changes to use system installed
boost instead of the included version of boost (boost_irods.tar.bz2)
which was no longer compatible.


git-svn-id: svn://irodssvn.ucsd.edu/trunk@5419 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Add logic so the admin can alias as other users when using GSI.  Also
add a log message (LOG_NOTICE level) for GSI logins similar to those
done for password logins.

Allowing the admin using GSI authentication to be able to alias as
other users via the clientUserName environment variable (as is
available with password authentication) was requested recently on
irod-chat as the 'Operations that require password authentication
(iCommands)' thread.  After checking into it, I found it wasn't too
difficult to implement and so went ahead and did so.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5420 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Check for data to read in the openssl buffer before calling select on the network connection.

Occasionally there is data left in the openssl buffers when the data read by the application doesn't match with the openssl encryption block sizes. In this case, calling select on the read file descriptor caused a hang because the data the application was waiting for was already in the buffer (this problem was observed when using SSL from the Prods API). The SSL_pending() call checks to see if data is in the openssl buffer, so we will only select() on the read if there is no data available.

Signed-off-by: Chris Smith <chris@distributedbio.com>

git-svn-id: svn://irodssvn.ucsd.edu/trunk@5421 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Corrections for MySQL for some of the PAM-related operations (user
acquiring a PAM-derived password, and iadmin rpp(remove PAM password))

Without these fixes those PAM operations would fail if the ICAT is
MySQL.

I discovered this problem after correcting a problem in the tinderbox
continuous test system so the MySQL test would run again (the Boost
runs take so long it threw off the scheme).  The 'irodsctl devtest'
(and so tinderbox) does detect this problem; MySQL issues a SQL error
so a CAT_SQL_ERR is returned when success is expected.

I'm looking at MySQL again because I'm working on integrating some
additional improvements in the iCAT (iPlant-provided) and wanted to
test them in MySQL too.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5422 ea137bd6-aa86-47b6-b90d-b101d9f19a90
add travis-ci.org configuration files - first attempt

git-svn-id: svn://irodssvn.ucsd.edu/trunk@5423 ea137bd6-aa86-47b6-b90d-b101d9f19a90
ICAT SQL performance improvement for very large scale sets of ACLs and
users with Postgresql.

These are changes to the SQL text for two SQL statements in the
chlModAccessControl function (used in, for example, ichmod) when
recursively updating the access-permissions rows (informally, 'ACL's)
on collections.  These changes were found to signficantly improve
peformance of these two statements in the iPlant instance where these
became slow due to there being a very large number of users and many
individual ACLs (rows) per collection.

The modification is to change IN to ANY and add a ARRAY( ) around an
expression for PostgreSQL (i.e. iPlant).

For Oracle and MySQL the ARRAY construct is not supported, so for them
I'm including the IN change but not the ARRAY.  I'm not sure if this
will improve performance for Oracle or MySQL under load, or not, but
it seems it might.  These pass tests for Postgres, MySQL, and Oracle.

These (for Postgres) were developed by iPlant staff, primarily Nirav
Merchant (with a little assistance from DICE).  Thanks!



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5424 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Added file/collection detection on the fly (at the cost of an extra query)


git-svn-id: svn://irodssvn.ucsd.edu/trunk@5425 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Make some corrections to the Tinderbox test description in preparation
for an addition.


git-svn-id: svn://irodssvn.ucsd.edu/trunk@5426 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Add a Tinderbox test for FUSE, run once a day.  Runs the test scripts
in clients/fuse/test developed by Hao. Assumes FUSE has been installed.

Also added one script, clients/fuse/test/test2.1.cleanup.sh, also run
by this, to remove some test files created by test2.1.sh.  This also
uses FUSE to do this.

As with all Tinderbox tests, if anything fails the test should
indicate an overall failure.


git-svn-id: svn://irodssvn.ucsd.edu/trunk@5427 ea137bd6-aa86-47b6-b90d-b101d9f19a90
In strict access-control mode, extend the checks made in the
general-query to handle items being checked (not just returned).

That is, in the general-query, when in strict access-control mode,
also add the access-check SQL when the where clause contains checks
using collection or data-object information.

The strict mode for access-control is when the core rule 'acAclPolicy'
is set to call 'msiAclPolicy("STRICT")'.  See core.re for a
description.  Default is not strict.

The previous strict-mode logic in the general-query code would add
access checks if columns in R_DATA_MAIN or R_COLLECTION_MAIN were
being returned.  This worked fine for queries done, for example, by
'ils' and many other internal general-queries, and many forms done by
users via 'iquest', 'imeta' and micro-services.

But, without the changes in this commit, users could find out if a 
data-object exists by name via queries like these:
  iquest "select USER_NAME where DATA_NAME = 'myfile'"

With this addition, queries like these, in strict mode, will return
only rows to which the user has read access.

This was noticed by Reagan in one of the classes he teaches.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5428 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Added support for MSO objects in msiPhyPathReg


git-svn-id: svn://irodssvn.ucsd.edu/trunk@5429 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Fixed race conditions in the directAccessFileDriver code.

When a transfer is done in parallel threads, the various calls to changeToRootUser()/changeToUser()/changeToServiceUser() might start happening in different threads at different times. Since the uid/gid of a process is shared by all threads, this would have the effect of a switch to root in one function be followed by a switch to a user in another, which would then cause the desired "root" operation to fail.

The fix is to use a mutex to sequence the operations that are done with switched context.

Signed-off-by: Chris Smith <chris@distributedbio.com>

git-svn-id: svn://irodssvn.ucsd.edu/trunk@5430 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Change default Postgres from 9.0.3 to 9.2.4.  Besides being the
latest, this also includes some security patches (see their web site)

I'm currently testing with the various Postgres versions that include
their security fixes, 9.2.4, 9.1.9, 9.0.13, and 8.4.17, and will post
to iRODS chat about this shortly.  So far I've confirmed that iRODS
works fine with 9.2.4 and 9.1.9.



git-svn-id: svn://irodssvn.ucsd.edu/trunk@5431 ea137bd6-aa86-47b6-b90d-b101d9f19a90