Talk:Distributed hash table

message: Second sentence of the article says: Each node is analogous to a bucket in a hash table. 
There is no expression "bucket" in the hash table article.
:It now reads 'array slot', but would 'record' be more appropriate? Nazlfrag 07:46, 1 March 2007 (UTC)
replies: text: The hash table article uses array index and (now) bucket as well as array slot.  Between these I don't much care; maybe we should look up with Knuth uses.  Record is probably not appropriate -- to me this implies an instance of a structure like a (key,value) pair rather than a position in the hash table's array. --Nethgirb 08:29, 1 March 2007 (UTC)
replies: text: Array index seems best, but I'll leave it for someone else to change. I agree with your rejection of record.Nazlfrag 11:54, 15 July 2007 (UTC)
replies: text: Bucket was the right term to use — it is a chain of records that share a common digest, as produced by the hash function. Array index merely refers to the fact that digests are used as indexes to the array of buckets (bucket pointers.) — 213.141.154.11 (talk) 14:46, 13 October 2008 (UTC)
replies: text: You are correct, damnit, I knew bucket was correct but I was thinking of making the page more accesible to those who didn't know. Luckily the page is much better now, but thanks for reminding me that bucket should have stayed as the proper description. Nazlfrag (talk) 15:42, 18 January 2010 (UTC)
replies: text: Agreed with Bucket being correct.  It could be clarified, but might be handy if we could use a tooltip or something to another article instead of cluttering up this article.  Like a CS oriented definition for what a bucket usually means.  I'd call a "bucket" as roughly synonymous with an extent or subset of a population of items.
Can this discussion be deleted?EternalStudent07 (talk) 05:15, 20 September 2018 (UTC)
thread_title:  Inconsistency 

message: * Give example topology
* Give example of typical use -- associate object with key, store object at owner of key, retrieve later by looking up owner.
* Discuss need to maintain O(log n) of the "right" neighbors for fault tolerance
* Reference all 4 original DHT papers
Nethgirb 01:53, 16 August 2005 (UTC)
thread_title:  To do list 

message: Hi everyone,
in the Structure section of this article it is said that the (k,data) tuple is moved to a node which is responsible for k according to the keyspace partitioning. Now, P2P systems are very dynamic systems. What happens if this node, where (k,data) is stored goes offline? Are there any replication mechanisms in place? And if so, how can duplicates and inconsistence or unavailability then be avoided?
Thanks for your answers.
Manfred
::Hi Manfred:  Most DHTs that store files use replication.  There are several methods and it would be a good addition to this article to have them described.  The most common method is to store replicas on nodes that are nearby in the keyspace to the owner of the key k.  So for example you can visualize the Chord DHT's keyspace as a circle, and each node is located at some point on the circle.  The replicas of a node v's data are stored at the r nodes clockwise around the circle starting at v, where r is the number of replicas, often 4 or 8.
::Maintaining consistency when data is mutable is trickier.  One technique is to have all writes handled by a single node, the owner of k.  If you want really strong guarantees (which as far as I know most DHT implementations don't provide), these are basically classical distributed systems problems not specific to DHTs; see two phase commit.
::Nethgirb 00:49, 9 February 2006 (UTC)
thread_title:  What happens if a node goes offline ? 

message: Can we please have a link to DEPOT?
:To keep the number of links manageable, I think we should only include links to projects which are notable, which probably means that the project has a significant number of users (either other researchers or regular people).  Does DEPOT fall into that category?  Also, if you were thinking that it should go in the "Applications employing DHTs" section, can you provide a reference showing that DEPOT uses a DHT? Thanks. --Nethgirb 20:12, 19 September 2006 (UTC)
thread_title:  DEPOT 

message: The Tangosol product has a DHT called "partitioned cache service" [] that meets the definition of a "decentralized distributed systems that partition ownership of a set of keys among participating nodes", and "route[s] messages to the unique owner of any given key." Channelsurfer 00:03, 7 November 2006 (UTC)
replies: text: Thanks for the link.  It does seem Tangosol splits up the data among nodes, but the partitioning of an abstract keyspace is one of the, er, key features of a DHT.  Another important DHT feature is that the information about who owns what keys is itself distributed: no one node knows everything.  It may be that Tangosol Coherence does these things, but they are not described on the page you gave or on the others I looked at.  In the absence of further documentation, I'd say we should leave it off the list; otherwise the list tends to become cluttered and less useful. --Nethgirb 09:46, 7 November 2006 (UTC)
replies: text: My name is Cameron Purdy, and I work at Tangosol. Sorry about the apparent previous link spam; I'll try to make sure that never happens again. Regarding DHT in our software, we dynamically partition the key space in a manner that can guarantee non-ambiguous ownership (and line of ascension), even during and after node failure. That is how the data are partitioned (since with key / value pairs, the value naturally goes with the key, therefore if the key domain is partitioned, the value domain follows respectively ;-). There is no global (or replicated) directory of keys; only the owner (and ordered backups) of a key knows that that key exists and what its value is. We had customers in production in June of 2002 on our DHT implementation, and at the time, I hadn't even heard of the term "Distributed Hash Table"; for lack of a better term, we just called it dynamic partitioning. As the explanation in the article of DHT is a bit cryptic (or I am simply not technical enough to understand it), I cannot say for certain that the algorithm that we use is the same as the one (or ones) suggested in this article; I would love to learn more about this particular part of the field and you can contact me at my first name at tangosol dot com and I will send you my contact info, etc. Having looked over the DHT article, the DHT implementation in Tangosol Coherence pre-dates (by at least a year) all of the articles and most of the implementations that you have referenced, and so I am quite surprised that you were not aware of Coherence. As for the academic initiatives in this space, I heard of one funded (a multi-million dollar grant) by the US government in 2003 to a professor who was going to "invent" such technology; I repeatedly emailed him to tell him about what our customers were already using in production, but he never responded. (8 Nov '06)
Hi Cameron, Looks like Tangosol deserves a mention in the article at least as related work.  I'm curious about how the process of discovering who owns a key works.  I could imagine a design like this:  one coordinator knows how the keyspace is partitioned among participating nodes, e.g., node 128.2.1.4 handles keys 0-100, node 68.4.35.30 handles keys 200-307, etc.  Then, to get a key's value -- or even just decide whether the key K has an associated value -- you ask the coordinator which node owns K, and then you ask that node directly about K.  Alternately, the coordinator's info could be replicated at all nodes.  Either way, there is no global directory of keys or their contents; but at least one node knows something about all the nodes in the system, in particular, who handles which keys.  This general scheme is used in a system called LH*, a 1996 paper from HP Labs  (see Sec. 3.5). Is this how Coherence works?
DHTs are even more distributed.  Not only is no one aware of all keys; in addition, no node is aware of all nodes.  Usually a DHT node knows only O(log N) of the N other participants.  So most queries end up being routed through a series of participants before they find the node that has the data (if any) associated with the target key.  Those O(log N) neighbors have to be chosen with care in order to ensure that queries do not involve too many nodes.  Since there is so little state at each node, DHTs can be very scalable, to a virtually unlimited number of nodes and high rates of node joins and leaves (e.g., each node participates for just a few minutes).  It is debatable how many applications actually need such extreme scalability, but there has been some specialized adoption as discussed in the article.
I would be interested to talk/email with you to learn more about how DHTs and Tangosol relate (certainly, the description in the article does need some improvement!) -- perhaps I'll drop you a note... --Nethgirb 00:47, 10 November 2006 (UTC)
thread_title:  DHT in Tangosol 

message: Under 'Properties' it says:
most commonly, Θ(log n) of the n participants (see below)
...which defines n as the number of participants, but what's the Θ signify?  Is it the name of some function, or a magic number to multiplied with log n.
And what base is the log?
Also later in the article, under 'Overlay network', there's a bunch of Degrees, that begin with 0.  
    Degree O(1), route length O(log n) ...
Should those 'ohs' be 'thetas' or vice-versa?
What I'd like to see is that any function is named as such (or linked to a relevant article), as soon as it's used, instead of being taken for granted.  Example:  
"the number of nosehairs on a goat may be approximately determined by Blackguard's function B(p), where p is how much money, in pennies, the goat cost."
  And the boldface Blackguard's function would link to a wiki that defines the function, but if the function has no article, it should be defined on the spot:  
"the number of nosehairs on a goat may be approximately determined by Blackguard's function, which is defined B(p) = (9999+p^2)/p + pi/p, where p is how much money, in pennies, the goat cost."
 
--AC 09:13, 7 February 2007 (UTC)
replies: text: See Big O notation.  "Big O" and "Big Theta" notation both tell you about the growth rate of a function, within constant factors, in this case as a function of n.  At the high level \Theta(\log n) means "about \log n", and O(\log n) means "about \log n or smaller" .  "Constant" here means "independent of n".  Since constant factors are hidden, the base of the log doesn't matter: whatever constant base you put there, the resulting value will only change by a constant factor.  You're right, it should be linked to the big O article at the first use. --Nethgirb 16:43, 7 February 2007 (UTC)
replies: text: Log to base 2. in Computing Science as computers use binary. FreeFlow99 (talk) 15:41, 15 November 2021 (UTC)
thread_title:  Θ(log n) : What's 'theta'? 

message: This article needs a serious cleanup, I have no idea what the author is trying to say. The concept of DHT should be explained in a more simpler manner. —The preceding unsigned comment was added by 68.146.236.11 (talk) 11:23, 12 February 2007 (UTC)
replies: text: Can you say more specifically which parts you found confusing? Thanks! --Nethgirb 18:15, 12 February 2007 (UTC)

text: Thanks, that helps --Nethgirb 22:11, 20 February 2007 (UTC)

text: That's because "DHT" is the generic term for the concept, as used in hundreds of different protocols/programs; it's not a functional/tangible protocol, it's just a building block for other systems. What BitTorrent in particular uses DHTs for is covered on BitTorrent protocol#Distributed trackers, not here. -- intgr [talk] 23:48, 6 December 2007 (UTC)

text: I have a degree in Computing Science, and I had to read this article twice.   A diagram would be a significant improvement.   A usage example would help. FreeFlow99 (talk) 15:37, 15 November 2021 (UTC)
thread_title:  Confusing 

message: Formatting for formula Θ(log n) is bad.  The article uses a single <math> element to code the formula.  This method is straightforward, and MediaWiki TeX should render the formula well.  Unfortunately MediaWiki TeX does not render the formula well.  MediaWiki TeX puts no space between 'log' and 'n'.
There is also a problem with the hyperlink for the formula.  The link to Big_O_notation should apply just to the 'Θ' character.
I played around with the formatting.  I came up with 2 ways to restrict the hyperlink text and extend the space.  The 1st way uses 2 <math> elements.  The 2nd way uses 3 <math> elements.  
Formatting             	Description
\Theta(\log n)         	The way it is now
\Theta(\log n)         	Restricts hyperlink to Θ, but still no space.  Requires 2 <math> elements.
\Theta(\log \mbox{ } n)	Some space, but not enough.  Requires 2 <math> elements.
\Theta(\log   n)       	Looks best.  Requires 3 <math> elements.
I favor the 3-<math>-element method.  It is the most complex, but I do not think it will matter.  The formula is semantically correct.  I do not think that anyone will want to edit the formula.  And the 3-<math>-element method looks the best.
Preferences?  Is there a simpler way to achieve extended spacing in the formula?
TT    23:00, 23 May 2008 (UTC)
replies: text: Maybe no one will edit it soon, but people still have to read the wiki markup, and people may create other formulas so it would add complexity there. Also, I think it's OK to have the Big O link for the whole formula.  After all, the whole formula is written in Big O notation. So I would suggest a variant on your third option: \Theta(\log \mbox{ } n) One math element, and I think it's plenty of space before the n. --Nethgirb (talk) 22:19, 25 May 2008 (UTC)
replies: text: Don't use math tags if you don't need them. What's wrong with the simplest form? Θ(log n)  -- intgr [talk] 00:32, 26 May 2008 (UTC)
replies: text: Nothing as far as I'm concerned, though to me Θ(log n) is just as good --Nethgirb (talk) 10:05, 26 May 2008 (UTC)
replies: text: Agreed; I was just making a point with regards to the overuse of . -- intgr [talk] 15:30, 27 May 2008 (UTC)
thread_title:  Formatting for Formula Θ(log n) 

message: Resource discovery activity involve searching for the appropriate resource types that match the user’s application requirements. Recent advances in the domain of decentralized resource discovery have been based on extending the existing DHTs with the capability of multi-dimensional data organization and query routing. Majority of the efforts have looked at embedding spatial database indices such as the Space Filling Curves (SFCs) including the Hilbert curves, Z-curves, k-d tree, MX-CIF Quad tree and R*-tree for managing, routing, and indexing of complex Grid resource query objects over DHT networks. Spatial indices are well suited for handling the complexity of Grid resource queries.  Although some spatial indices can have issues as regards to routing load-balance in case of a skewed data set, all the spatial indices are more scalable in terms of the number of hops traversed and messages generated while searching and routing Grid resource queries.
-- moved in from peer-to-peer.    M   19:12, 14 June 2009 (UTC)
thread_title:  Some content 

message: The current discussion of history is misleading. It could be read as implying that Cord, CAN, Pastry, and Tapestry invented DHTs in 2001, but there were earlier systems. I propose revising the "history" paragraph as follows:
Old text:
The first four DHTs—CAN, Chord,Hari Balakrishnan, M. Frans Kaashoek, David Karger, Robert Morris, and Ion Stoica. Looking up data in P2P systems.  In Communications of the ACM, February 2003. Pastry, and Tapestry—were introduced about the same time in 2001.  Since then this area of research has been quite active. Outside academia, DHT technology has been adopted as a component of BitTorrent and in the Coral Content Distribution Network.
Proposed new text:
The first DHT implementation—the Beyond Browsers systemRenu Tewari, Mike Dahlin, Harrick Vin, and John Kay.Beyond Hierarchies: Design Considerations for Distributed Caching on the Internet.  University of Texas at Austin Computer Sciences Department Technical Report number TR98-04, February 1998.—was introduced in 1998 and was based on Plaxton, Rajaraman, and Richa's algorithm.C. G. Plaxton, R. Rajaraman, and A. W. Richa. Accessing nearby copies of replicated objects in a distributed environment. In Proceedings of the 9th Annual ACM Symposium on Parallel Algorithms and Architectures, Newport, Rhode Island, pages 311-320, June 1997. In 2001, four systems—CAN, Chord,Hari Balakrishnan, M. Frans Kaashoek, David Karger, Robert Morris, and Ion Stoica. Looking up data in P2P systems.  In Communications of the ACM, February 2003. Pastry, and Tapestry—ignited DHTs as a popular research topic, and this area of research remains active. Outside academia, DHT technology has been adopted as a component of BitTorrent and in the Coral Content Distribution Network.
Md4567 (talk) 14:42, 2 February 2010 (UTC)
replies: text: Looks good; Be bold in updating pages. If someone disagrees, they will create a discussion themselves and revert (WP:BRD) -- intgr [talk] 18:40, 2 February 2010 (UTC)

text: I disagree that Beyond Browsers was the first DHT, because its architecture is fundamentally hierarchical (as is clear from the referenced paper) and thus it is not decentralized, and fails to meet the first of the three DHT properties outlined in the article. Javalangstring (talk) 22:05, 6 July 2010 (UTC)
thread_title:  DHT History 

message: Okay so all I'm hearing is that golden word "decentralization" every sentence but I still don't understand how I can connect to the swarm without a central node telling me what IPs to connect to.
I understand this system is decentralized, that is fine, but how a completely decentralized system can be organized is my question. If I run, lets say a P2P program that uses DHT, and I search for the word "london", which IP will my computer first connect to and why? How will my computer know which IP out of the 4 billion to first connect to and request the hashes of that keyword I searched for? I really don't get it.142.59.42.197 (talk) 04:34, 11 April 2011 (UTC)
thread_title:  So how does it work exactly? 

message: Hello fellow Wikipedians,
I have just modified  on Distributed hash table. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive https://web.archive.org/web/20070522060750/http://www.sics.se:80/~ali/thesis/ to http://www.sics.se/~ali/thesis/
*Added archive https://web.archive.org/web/20101204111423/http://www.tribler.org:80/trac/wiki/Khashmir to http://www.tribler.org/trac/wiki/Khashmir
When you have finished reviewing my changes, please set the checked parameter below to true or failed to let others know (documentation at ).
Cheers.—InternetArchiveBot (Report bug) 01:15, 14 December 2016 (UTC)
thread_title:  External links modified 

message: In the section 'Overlay network' is a table of various implementations of the math to determine max degree and route length. As explained just above the table, the implementation determines the number of neighbors to 'handle' and maximum number of hops/route length.
It would help immensely with understanding the differences, if an example of the resulting math was shown, either in columns added to the right or separately below. I.e., if a network has 1 million nodes, what number of neighbours and max hops would that result in if using "O(\log n) O(\log n)" versus "O(\sqrt{n}) O(1)". -- Katana (talk) 07:17, 15 January 2017 (UTC)
thread_title:  Please add example with numbers to 'log(n) math' section 

message: Hello fellow Wikipedians,
I have just modified 2 external links on Distributed hash table. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive https://web.archive.org/web/20150926070618/https://iris.pdos.csail.mit.edu/MITPressRelease1.doc to https://iris.pdos.csail.mit.edu/MITPressRelease1.doc
*Added archive https://web.archive.org/web/20040910154927/http://www-db.stanford.edu/~manku/phd/index.html to http://www-db.stanford.edu/~manku/phd/index.html
When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.
Cheers.—InternetArchiveBot (Report bug) 12:16, 11 September 2017 (UTC)
thread_title:  External links modified