Docker + AV on the host I've dug through some history of this site (which inevitably means someone will post how I missed it) and didn't find anything directly relating to my question.

What has been the typical solution when it comes to running docker but still trying to meet security requirements?

The particular use case is that we have dedicated container hosts and following Docker's guide on AV, have certain directories excluded. We still have issues with containers somehow getting roped into on demand scans, so we turn that off. Still have issues, so we exclude more directories. In the end we've disabled parts of the AV solution and excluded any directory users would be writing to, which - in my opinion - makes the AV solution ineffective. So I'm curious what other companies responses have been, whether it's technical (different vendor, settings in AV, etc) or policy driven (not requiring AV on Docker hosts).

With a Docker container host, I think the best question to ask in terms of deciding your policy is "what threats does Anti-Virus address on this host"?

A-V is designed to catch malware running on a host, so the question becomes, how can malware be introduced onto that host.  For Docker containers, that would either come via the images which are used to create the containers, or via data uploaded to applications running in the containers on the host.

For the first risk, a better solution would likely to be to A-V scan the images as part of the build process, alongside other security checks like vulnerability assessment.  Assuming you're using common CI/CD pipeline tools like Jenkins, it should be possible to integrate an A-V scanner into your pipelines.

For the second risk, if you're thinking about files uploaded as part of the application's operation, then you could target A-V scanning at purely the directories used by the applications to receive uploaded files.  As Docker containers are generally ephemeral, you would likely be using external volumes for storage of uploaded files, which makes it easy to target A-V at specific directories, as opposed to scanning the whole host.

The only other scenario that springs to mind would be where an attacker was able to compromise an application running in a container and uploaded tools which may be caught by an A-V scanner into the running container.  For this you would likely need to run a scanner which targeted the Docker directories, I guess it would come down to your risk assessment as to whether that was a risk you considered serious enough to warrant the performance hit of allowing an A-V scanner to run in those areas.


Are there any situations when one can only mount a passive MITM? This came up while discussing Web & insecure HTTP - Using RSA for encrypting passwords on the client side

Is there any such situation possible when requesting an HTTP page where an attacker is able to read all the communications but is not able to modify/spoof the communications?

Till now I've worked with ARP spoofing, which lets one modify packets if necessary. AFAICT if the network is willing to send you packets which aren't yours, the network will accept packets coming from you that appear to come from someone else.

Passive-only attack scenarios tend to be rather specialized. Mostly, they involve radio links; the attacker can play with a homemade reception antenna, but is not rich enough to build an emitter which can drown out the genuine signal. This rarely applies to the Internet; this is more relevant to military on-field communications.

It could be argued that defeating passive attackers forces said attackers to go active, making them potentially easier to detect and retaliate upon. That's a rather weak argument, though. There again, it more applies to spy networks than to Internet business.


CSR with only public key with openssl So I have been doing my research and haven't found anything specific enough to my problem. As the question title states, I want to make a CSR with only the public key.
What I have been able to do is generate a CSR with the information of my choosing along with a new keypair. And then use something like this to force it to use the public key I want when making the certificate.

openssl x509 -req -in mycsr.pem -force_pubkey mypubkey.pem -CA dumyCA.pem -CAkey -dumyCA.pem -out mycert.pem

After this I take the outputted certificate and change its attributes to associate it with a private key. This all works nicely. The problem with this is that if I were to take the said csr, and submit it to a CA they would return to me a certificate that used the key that was generated when the csr is generate. Thus my question.

EDIT:
So I have a key-pair, but my private key is on an HSM module that doesn't have export capabilities. I am able to get a .pem file of the public key. And after making the certificate with the command I gave I run it through a program that associates it with the private key. So my new question is, using openssl API will I be able to create my own csr? So I can sign it with the private key without having to have it in a file, and associate it with the public key I have.

The Certificate Signing Request, as generated by the command :

openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key

will include your public key. This is mandatory as per the PKI process. The CSR, containing your entity information and the public key is sent to any Certificate Authority you like for a request of certificate (hence the CSR name). The CSR is signed using the private key that is linked to the embedded public key.

The CA, after assessing your identity, will basically sign the CSR to create a valid certificate for you. This is the certificate you will send to clients, e.g. when they connect to your website.

The private key in your privateKey.key file will be used by your webserver to decrypt messages sent to it. This private key should never, ever leave the server.

You can check the most common openSSL commands for more information.


How is Insufficient Attack Protection a Definite Threat/Risk to an Organization? Recently, OWASP introduced two new set of categories as of 2017, April - to it's OWASP Top 10:

Insufficient Attack Protection
Unprotected APIs

I understand, Unprotected APIs does have an immediate risk which involves proving a huge attack surface along with possibilities of data leakages, however, I fail to understand how Insufficient Attack Protection is any threat or a risk for a category?

Improving my focus, I would summarize quoted from scamdemy:

  Insufficient Attack Protection refers to the inability to detect, prevent and respond to various kinds of attacks against the application as a whole. This – due to the large number of unaudited third-party components that may contain critical vulnerabilities – necessitates the use of generic security tools such as intrusion detection systems (IDS), and web application firewalls (WAF) that can identify an ongoing attack such as SQL injection. It focuses on the consequences instead of the root causes of the weaknesses.

Does this imply to having WAF set-up in direct connect to having a great attack surface area without the presence of Firewalls? If absence of a component is an immediate categorization need on OWASP Top 10. Not sure, how other's aren't affected by the same?

e.g. By not having WAF, certain levels of Injection(s) will be evident given that their is a flaw in application code.

Do i presume, this is a move for the security audit team to market their Firewall products keeping OWASP Top 10 as a reference? Or was it really necessary technically?

"Insufficient Attack Protection" is a horrible choice of words, but I don't have a suggestion on how to improve it.

I've overseen applications which were missing really basic detection abilities and it's frustrating trying to communicate the urgency of some kind of response to the application team.

Imagine an application with:

no protection against brute force password attacks,
no logging of login attempts,
no logging of session initiation or completion,
no logging of attempts to manipulate expired sessions etc.

I found myself writing brittle and awkward Snort signatures to extract basic logging data and tuning application firewalls to try to make up for missing behaviours while application teams simply said "meh, it's infosec's problem".

IPS signatures can't know "is this a valid user?" or "is this session active?" or "how much data has this person used today?", but application logic may have access to this information, or maybe just having the sessions logged would mean the SIEM could perform the logic.

Regarding scamdemy's article, I think he has it wrong.  This has nothing to do with third-party components.  Many of the security articles coming out on the proposed 2017 A7 seem to be knee-jerk reactions to pumping out a response to OWASP's new Top-10, it gets attention I suppose, but I don't think the authors have given their response careful thought.

The external links inside the A7 cheatsheet are far, far more descriptive:

http://cwe.mitre.org/data/definitions/778.html
http://cwe.mitre.org/data/definitions/799.html
http://projects.webappsec.org/w/page/13246938/Insufficient%20Anti-automation

IMHO, A7 needs some tweaking, I think "patching" references should be removed.  Infosec can do virtual patches or dev processes can patch, but the references here detract from what I believe is their core message.


Does an insecure computer on my home wireless network threaten other machines? I have a DD-WRT based wireless network with strong password security.  However, other machines on the network may not be as protected as mine.  If these machines are compromised, need I fear the attacker can more easily hack my machine, having gained access to another machine on the wireless network?  We do not share domains.

It does indeed slightly facilitate a remote attack on your machine, since an attacker would be able to send you packets from your local network, rather than having to rely on NAT traversal from the router you use to connect to the Internet. You would still need to have network-facing services running on your machine, such as SMB/CIFS, Telnet, other protocols for local peer discovery (e.g. Bonjour), or the attacker would need to attack certain layers of your OS's network stack (since they can send you packets directly, although that's quite rare).

What you should be slightly more worried about are possibilities for ARP spoofing, DNS spoofing, and so on. If you start getting invalid certificate errors or seeing unexpected content whilst using the Internet on an untrusted local network (usually public WiFis, but a private WiFi with infected machines would work just as well) then you should be worried.


Can BREACH be thwarted by simply adding a sort of "salt" into the page being compressed? Django supplies GZip middleware, but the docs issue a stark warning about BREACH, which I had, hitherto, forgotten about. The first thought I had was that I should be able modify the middleware to first add a random amount of random characters into the request body before GZipping it, and render BREACH useless. Is this correct, or am I completely misunderstanding BREACH?

If you have always the same amount of random data the attack will probably still work in the same way. If you instead add a random amount of random data the original attack might not work any longer but I think it can be modified by simply trying again and again in the hope that the same amount of random data is added often enough. Which means it will slow down the attack only and how much depends on the amount of random data you add. For example if you add between 1 and 10 random characters at the top of the file it will probably only slow down the attack by a factor of 10. If you add 1..100 characters the slowdown will be factor 100.

A better approach would be to create some random value and modify the CSRF token so that it will consist of this random value and an XOR or with this value with the original CSRF token. This way the CSRF token will change all the time inside the HTML so that BREACH will no longer work.  And the application can easily get the original CSRF token back.


Multiple wp-login.php attack on shared hosting I am hosting more than 20 WordPress website on a single server. The server has other cms based websites too. For better security, I have used CloudLinux CageFS which encapsulates each customer. Even if one WordPress site gets hacked then there is no way that hacker will be able to find other users in the server. S/he can't even see other users by reading /etc/passwd.

Even though I am using such security I see lots of attack on WordPress websites. My question is how is it possible. I know there are sites like yougetsignal
which can disclose sites hosted in same IP address but in my case it also does not show other websites. How is the hacker able to find other WordPress sites on my server and how is s/he attacking those sites?

The processes looks like below:

# ps aux| grep wp-login
user1 10904  0.1  0.1 278288 44244 ?        Ss   12:19   0:00 lsphp:/home/user1/public_html/wp-login.php
user1 10905  0.1  0.1 278304 44280 ?        Ss   12:19   0:00 lsphp:/home/user1/public_html/wp-login.php
user2 12171  0.1  0.1 394312 46396 ?        Ss   12:20   0:00 lsphp:ome/user2/public_html/wp-login.php
user2 12614  0.2  0.1 394536 46480 ?        Ss   12:21   0:00 lsphp:ome/user2/public_html/wp-login.php
user3 12687  0.1  0.1 382632 35524 ?        Ss   12:21   0:00 lsphp:home/user3/public_html/wp-login.php
user3 13256  0.0  0.0 382240 27964 ?        Ss   12:21   0:00 lsphp:home/user3/public_html/wp-login.php
user4 14335  0.1  0.0 380504 20460 ?        Ss   12:22   0:00 lsphp:/home/user4/public_html/wp-login.php
user5 14718  0.1  0.0 380296 18408 ?        Ss   12:22   0:00 lsphp:/home/user5/public_html/wp-login.php
user6 14735  0.0  0.0 379564 16376 ?        Ss   12:22   0:00 lsphp:/home/user6/public_html/wp-login.php
user7 14750  0.1  0.0 380272 17376 ?        Ss   12:22   0:00 lsphp:/home/user7/public_html/wp-login.php
root     15078  0.0  0.0 112712   976 pts/0    S+   12:23   0:00 grep --color=auto wp-login

When I checked the access log then I see request to wp-login.php file from different IPs, connections are unique as request from same ip is no more than twice. Is not this an attack? The access log looks like below:

108.169.181.185 - - [22/Sep/2019:14:14:29 +0200] "GET /wp-login.php HTTP/1.1" 200 11911 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
108.169.181.185 - - [22/Sep/2019:14:14:34 +0200] "POST /xmlrpc.php HTTP/1.1" 403 9 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
2403:cfc0:1007:100::10 - - [22/Sep/2019:14:14:34 +0200] "POST /wp-login.php HTTP/1.1" 200 9948 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
2403:cfc0:1007:100::10 - - [22/Sep/2019:14:14:37 +0200] "POST /xmlrpc.php HTTP/1.1" 403 9 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
178.62.98.196 - - [22/Sep/2019:14:15:33 +0200] "GET /wp-login.php HTTP/1.1" 200 11905 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
178.62.98.196 - - [22/Sep/2019:14:15:34 +0200] "POST /wp-login.php HTTP/1.1" 200 11905 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"

Although you don't provide this level of detail in your question I'll assume that the WordPress sites you host are publicly accessible and that they either have their own domain name or that they are linked from somewhere else. What you might see then is not that an attacker specifically attacking your IP address, but an attacker addressing lots of publicly known sites. Only you have a limited view into this attack, i.e. you only see the attacks targeted at your specific IP address and thus assume that you are the main target of this attack.

Of course, there might be other ways the attacker might specifically address all sites on your IP address, like having access to backups of your server configuration, to passive DNS data or by using services like domaintools. In general these information are not secret so they cannot be relied on to protect sites.


Password and Generated number - How does it work? In my company, they don't allow accessing the company email or servers from outside the company network.

To access the email from outside the network, you have to use: Username, password and a number generated from a small number generator.

How does this verification method work?

There are several methods for such tokens. One of them is HOTP: the token and the server both share a common secret value, and a counter; from the secret and the current counter value, a one-time password can be generated (the token displays it, the server verifies that the entered password matches that which was expected).

Some tokens also include the current date and time in the process, so that the one-time password is also limited in time: this supposes that the user pushes the button on the token and immediately enters the displayed password (the use of time prevents the user from generating some OTP in advance and writing them down on a piece of paper). The usual standard for that is TOTP.


Where should I store a refresh token I'm building an oauth 2.0 protocol. I'm wondering how the refresh token works exactly.

My understanding is that the use of a refresh token enable short lived access token and therefore limits the vulnerability of those access tokens. Great so far.
Once an access token expires, you somehow use the refresh token to get a new access token.

I'm wondering how that last part happens exactly (without interrupting the user flow by logging them out):

to secure the transaction, should it be required that the app id and app secret are sent along the refresh token, to ensure the identity of the requester?

if so, it sounds like you don't want to store an app secret on a client (web or mobile app or other) so how can you do that without logging out the user?
if not, what is the point of the refresh token if you can't use it to verify the requester identity? And is it ok to store the refresh token in a client?

Compromise of refresh token would cause more harm than compromise of access token.

You can employ various techniques to make it more secure.

You can get the idea by learning about one of the standard technique which is recommended for "The OAuth 2.0 Authorization Framework". Please refer the RFC.

You can separate the concern of access and authorization by maintaining separate resource server and authorization server. You can use only access token (and not refresh token) to access resource. You can use only refresh token (and not access token) to communicate with authorization server. This mitigates the risk of refresh token getting compromised.

Secondly, it is easier to detect if refresh token is compromised.
Let's say a refresh token is comprised and is used to generate new access tokens. Once this refresh token is revoked, all the access tokens generated already and latter will get invalidated. As such only 1 party will be able to access the resources using access tokens generated using the single valid refresh token. If multiple users are participated, they will go on invalidating each other. In the process detection of refresh token compromise gets easy.


Is pull inherently more secure than push? Users of a service are waiting for something to happen - a money transfer for example.

An attacker might forge an email notification, that executes an XSRF attack.

Alternatively, I could leave it up to the user to check back as and when they want to, by visiting a web page. Over https, the notification they receive is encrypted and signed as being genuine.

My examples are biased against the flaws of email. There are lots of different types of push and pull messaging systems.

That said, I am curious to know if there is something inherently more secure in pull based designs?

I don't think there is a big difference between the two models. If you implement the "push" model, which I'll call the "notification" model, the user can still log into the site before getting the notification. If you implement the "pull" model, which I think of as the "polling" model, an attacker can still email the user with a forged notification, effectively converting your no-notification polling model into a notification model (the attacker would need to know that a notification is expected to achieve this effectively). So, either way, you are basically supporting both models.

In the end, your user must be sufficiently sophisticated to avoid clicking on bad links in their email, and generally avoiding going to incorrect websites. If they aren't, then they are open to phishing and other attacks. Implement 2FA, consider something like U2F, train your users about phishing attacks, and make sure that you're support team knows how to respond to problems.


Is Linux really that secure, with only root password denying external access? While I was searching online for information about Linux security, the most typical explanation was: Linux is secure, because the root password is required to access the kernel and install new applications - therefore external malicious software can't do any harm as long as the administrator is the only person to know the password.

OK, that sounds good. But when a password is the only thing that stands between restricted access and total control of the system, is the system really that secure? By that I mean all kinds of tricks hackers think of to access systems, and particularly to reveal data (passwords).

"Linux" (as some aggregate of all the installations) typically has quite a bit more than just a password denying external access.

First, there's a uniform set of discretionary access controls: read/write/execute permissions, for user/group/everybody else. Traditionally, these permissions are actually used, rather than ignored and/or worked around. Additionally, some subset of installations have SELinux installed, configured and working, so that finer-grained, access control list style of permissions is enforced.

Second, servers usually run as a designated special user. NTP processes run as as user "ntp". Web server processes run as a user "http", MySQL databases runs as a user "mysql", for example. The descretionary access controls described above almost always prevent the NTP user ID from doing much more than reading some of the HTTP user ID's files.

Third, the software installed base is highly fragmented. There's a huge number of different distributions. After that, not every installation runs Apache HTTPD, or sendmail SMTP server. There are alternatives, and there's usually only a plurality of installations with a given server. Versions of software are also highly fragmented. With every distribution compiling and maintaining its own choice of web server, it's very, very rare for two installations to run a server that has the same bugs, or even the same compilation options.

So, for instance, someone going after a Linux machine via WordPress password guessing can maybe guess the WordPress password. That might get the attacker something running as user "http" or "apache". Bad and horrifying as that might be, it's not everything. The "http" or "apache" user almost certainly can't overwrite very many files at all, only HTML and what have you in the DocumentRoot directory. It would take another leap, guessing the "root" password for some distributions, or exploiting a local privilege escalation, to get to some kind of universal file access.

This really is multiple layers, but note that it's mainly by culture and tradition, and it's also a sort of "herd immunity". It's always possible that some combination of exploits would yield root access on a given system, but that combination probably wouldn't apply to very many other systems.


Securely compile foreign code and run it on a server? I would like to set up a server that can receive source code directly, compile/interpret it and execute it.

The code could be C#, Java, C++, C, Javascript, PHP...

There will obviously be security concerns, I would like to know if you have any recommendations to prevent the server from being compromised?

From what I look on the internet, Docker seems to be a good solution. Is that correct?

I would like each code to be independent, so if malicious code destroys it's own virtual machine, it does not matter.

To explain more in detail, I think of a game of programming, the players will write their codes, locally on their machine and send them in an arena to compete for example in 1 vs 1 with, in between, the engine Game.

The idea would be to have 3 programs, for a game in 1 against 1 (2 players and the arena) the information exchanged between each of the programs would be characters strings I would also like to the player program to be compiled on a server (in the vm docker for example).

I would like the code to run on the server side so that each player has the same computing power.

What you're looking to do, if I read your question correctly, is to isolate possibly hostile code to be compiled and run on a server you control.

So you need some kind of sandbox/isolation to do that.

The question you need to think about is "how much isolation do I need"

Docker will provide reasonable isolation but it still exposes the Linux kernel on the host (all containers, by default, share the same Linux kernel) so an attacker could try and exploit an issue there.  Also attacker code could do things like scan the local network to try and find other things to attack.

Some possible options to improve the isolation provided by base Docker.

1) Drop capabilities from the running containers using the --cap-drop=ALL option.  This reduces the privileges that the container has and makes it harder to break out

2) If the running container doesn't need network access use --network=none so it can't attempt to make connections out to other machines.

3) Don't run the containers as root, make sure you set a specific user for each container.

You could also isolate each task inside a different Virtual machine, although there's a resource hit over Docker for that.


What's a practical example of how volatile information can be preserved in a digital forensics investigation? I'm taking a study course on CCSP and am going over a section on the digital forensics investigation process.  In particular, the instructor discussed the following:

I'm curious on two bullet points here:

Capture an accurate image of the system.
Work from volatile to persistent evidence.

Both of those make sense by themselves but I'm wondering if/how it's possible to capture an image of a system INCLUDING volatile evidence.

Wouldn't I have to shut the system down (or reboot) in order to collect an image?  If so, wouldn't I lose a large chunk of volatile information (e.g. running processes, memory, etc)?

From a practical perspective, how can volatile evidence be preserved?

Wouldn't I have to shut the system down (or reboot) in order to collect an image?

No, you can use a validated memory capture tool to collect the volatile evidence in RAM.

  If so, wouldn't I lose a large chunk of volatile information (e.g. running processes, memory, etc)?

Maybe, it's really hard to say. Research at Princeton University has shown that RAM fades rather than disappears. Cooling allows examiners more time to collect the volatile data. There are a few other concerns with pulling the plug:

Encryption. Powering off the machine could encrypt files that are unencrypted while the machine is running, meaning a loss of evidence.
Damaged data. Turning off the machine could render data unreadable.
Unrecorded evidence. Evidence may not be recorded unless the machine is properly shut down.

  From a practical perspective, how can volatile evidence be preserved?

You need to take a series of steps:

Once you begin, make sure you work uninterruptedly, otherwise you invite mistakes.
Gather your materials: report forms, pens, memory capture tools, etc.
If you need to press a key to wake the machine up, document it.
Note the date and time as it appears on the computer.
Record visible icons and running applications, and document running processes. This could help identify malware. You can access this information via the task manager.
Use a validated memory capture tool to collect the volatile evidence in RAM.
Properly shut down the machine to allow any running application a chance to write any artifacts to the desk, which will allow you to recover them later.

You can read more about this process in The Basics of Digital Forensics: The Primer for Getting Started in Digital Forensics by John Sammons.


Security implications of the contents of /proc/self/environ in LFI attacks During penetration testing on a website, the tests have found a local file inclusion vulnerability in an old wordpress plugin that was installed. An attacker can exploit the LFI vulnerability to include /etc/passwd and also the index page, however when including /proc/self/environ, all the attacker sees is this:

CONTEXT_DOCUMENT_ROOT=/home/[website]/public_htmlCONTEXT_DOCUMENT_ROOT=/home/[website]/public_html

Isn't /proc/self/environ supposed to show other information? Such as your user agent? If so, why is it that for this website, all it shows is CONTEXT_DOCUMENT_ROOT?

I know that when an attacker includes /proc/self/environ and it does show their user agent, they can use tamper data for example to change their user agent to PHP code and upload a shell for example or open outbound TCP connections etc.

So, basically since this is all /proc/self/environ shows, does that mean that /proc/self/environ is basically safe from attackers? Or can they somehow manipulate this to show their useragent and run PHP code? (Forgive me if this is a stupid question, I am somewhat new to security. Also, ignore the elephant in the room: The actual LFI vuln will be fixed so it isn't vulnerable at all, but I am still curious about my /proc/self/environ question.)

/proc/self/environ contains the environment of the process. In this case, only the CONTEXT_DOCUMENT_ROOT seem to be present (there would be a \0 between the two copies).

This would be possible. If your php application isn't run as a CGI, there's no need for the HTTP_ variables to appear there. The other SAPIs (such as FastCGI or an apache module) receive them in a different way.

However, even if we are only seeing the server environment, and it is not used to pass parameters to the application, it seems peculiar that typical variables like PATH or HOME don't appear there. And that the same variable appears twice doesn't make much sense, either.

Why don't you upload a basic php file to print the contents of the /proc/self/environ?

  I know that when an attacker includes /proc/self/environ and it does show their user agent, they can use tamper data for example to change their user agent to PHP code and upload a shell for example or open outbound TCP connections etc.

Not at all. In order to run code, the server would need to eval() the included User Agent, which would be quite silly.

The environment contents may be useful for gaining more info about the server (such as paths) for further attacks, though.


How are salts safe in the database? I have read many and many of this type of question on the security stack community before I asked this question, but one thing that made me mad about salts and saving it in database is this ... I will put myself in the attacker's position.

If I hack into a database and find a table called for example ( users ) and I see a field named ( salt ) and another one ( password or hash ) I can easily - with no effort - know that the salt is used to produce this hash or this password.  So I will take the salt of the first record for example and make a dictionary of all hashes of all English words ( rainbow table ) and then I'll use the salt that I got from the first record and try hashing it with each hash I got in the rainbow table and I will get the salted hash easily at the end ........ so please why do all people say on SO or the Internet say that putting the salt in the database is good practice or safe? .. I don't see it as safe and please correct me if I am wrong .. thanks in advance.

I have postponed my project because of this issue.

The purpose of salting is, that one cannot build a rainbow table to get several passwords at once.

Without salting: An attacker could search the internet for precalculated rainbow-tables and find the passwords with no effort.

With a constant salt: The attacker has to build one rainbow-table for this specific salt, and can then get all the passwords with this single rainbow-table.

With a unique salt: If each password get its own salt, the attacker has to build a rainbow-table for each password. In this case brute-forcing is faster, it doesn't make sense to finish the rainbow-table after finding a match, because you cannot get other passwords anyway.

So you see, even if the salt is known it fulfills its purpose. If you want to add a secret to the process, you should instead use a server side key and encrypt the already calculated password hashes. Do not mix up salting with other security measures.


Fake UserAgent visitor attack? Not sure if I should report it here, but within my website I collect each request in DB, and from time to time view these records. Among data collected are user agent, reqested url, referrer (i.e. previous) url, time, and others.

Today I found a following user agent bot (which seems to hide this):

(Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko

IP:

181.114.49.183

Most of requests are within 1-2 seconds intervals, and the most interesting is what urls it wants my server to check:

| /wp/wp-login.php        |
| /blog/wp-login.php      |
| /site/wp-login.php      |
| /cms/wp-login.php       |
| /section/wp-login.php   |
| /wp-admin/wp-login.php  |
| /wordpress/wp-login.php |

(from my DB)

Referrer in each cases is null (meaning there was no referrer; it expected to find a webpage at this uri).

Why a bot may possibly want to check if there is a wordpress installed on a webpage, and what is its url (my website doesn't use wordpress, or any other CMS)? To me, it's like searching for a possible way to get into an admin panel of the website. And I do have an admin panel which I created myself.

Should I do anything about it?

Every computer with a public IP gets this kind of attention permanently. There's nothing you can do to stop it (I once tried to complain to the provider owning the offender's IP, never got a reply and gave up). What you can do is to make sure you're well protected against a possible attack (this bot seemed to look for WordPress, but there are others looking for Apache, SSH, you name  it). A few rules:

Expose as few services as possible. If you don't need SSH, FTP, etc.,  disable it.
For the services you expose (the web server in your case) make sure you install security patches regularly.
If your service has some form of authentication (like WordPress admin page), be sure to pick a strong random password. Online bots usually check for default passwords and extremely weak combinations like root/r00t, but I wouldn't risk using any dictionary word, or anything shorter than 12-16 characters.
If you want to stop wasting resources on people who try to guess your password (assuming you have a good password) you can install Fail2Ban which bans an IP address for 10 minutes after 6 failed login attempts, rendering password guessing scripts impractically slow. Of course, you can configure the ban delay and the number of attempts to your liking.
For services which are intended for a specific group of users (you, your company etc), you can also use other techniques like port knocking and limiting access to IP ranges you are likely to use to access your services (your country only, your ISP, your work ISP etc.).


Diffie Hellman implementation- NodeJS Diffie Hellman is a key exchange algorithm where client and server both generate public and private key, exchange their public key and combine this key with his own private key to generate same secret key.
But, here is a confusion in the implementation. Here is the code...
const crypto = require('crypto');
const express = require('express');
const app = express();

// Generate server's keys...
const server = crypto.createDiffieHellman(139);
const serverKey = server.generateKeys();

// Generate client's keys...
const client = crypto.createDiffieHellman(server.getPrime(), server.getGenerator());
const clientKey = client.generateKeys();

// Exchange and generate the secret...
const serverSecret = server.computeSecret(clientKey);
const clientSecret = client.computeSecret(serverKey);

First of all, server create an instance of DiffieHellman class to generate key.
But, client need server's prime (.getPrime()) and Generator (.getGenerator()) to generate another instance of DiffieHellman class to generate key.
So, server need to pass the value of server.getPrime() and server.getGenerator() to the client.
What happen if any middle-man-attack rises in this time? Because, if somehow hacker get this two things then they can also generate same secret key. (-_-)
Any solution? Think this system without TLS.

You are assuming that generateKeys is a deterministic operation so that a attacker knowing the DH parameters will generate exactly the same keys. But this is a wrong assumption. The attacker will end up with different keys even when using the same DH parameters since randomness is involved when generating the keys and the attacker cannot replicate the randomness used by the client.


Security of LastPass together with YubiKey I'm looking at password manager solutions and came across LastPass. I see that they also support two-factor authentication using YubiKeys. How secure is this combination for password management? What are the "weak links" in this scheme that could be targeted by an attack?

The answer everyone hates: it depends on your threat model and risk appetite.

What passwords are you protecting in Lastpass?
Are you storing the whole password in there or a unique value to which you add a passphrase?
Who are you concerned would want your passwords? Opportunistic attackers or targeted governments / organized crime?
How strong is your master password?

Software vulnerabilities can exist. Lastpass has had a XSS vulnerability and a suspected intrusion recently. So yes all software can have vulnerabilities.

Yubikey as @this.josh states could also be vulnerable. After all if RSA got hacked and the attackers were able to use this to get into military contractors then no two factor mechanism is invulnerable.

Refer a sample attack tree for defeating two factor:

Here is a broader set [PDF]: http://www.redforcelabs.com/Documents/AnalyzingInternetSecurity.pdf

The question is are the risks acceptable to you?

Using a password manager is better than not using one and is a simple, cheap solution to improve the security of virtually any application/service you need a password for.

Using Yubikey and a strong master password greatly improves the security of whatever you store in Lastpass. The whole point of two factor is that even if one factor is compromised they still require the other. If you or the service discovers the compromise this gives you time at a minimum.

Do a quick threat model, understand your risk appetite. No system will be invulnerable but you may find the advantages to using Lastpass + Yubikey outweighs the risks for you.


Does using TLS, SSH and SSL in a connection makes it more secure? If I use TLS as the protocol to send SSL encrypted packages through a SSH tunnel would this connection be safer than using any of the tools alone?

** EDIT **
Suppose I would TLS(SSL(SSH(data))) (thanks forest), a MySQL database connection for instance, would it be safer than just SSH(data) or TLS(data)? This question popped up at the uni and we discussed it for some time with mixed results.

PS.: If this sounds like a beginner question it's because it is.

In general I would say No.

For almost all cryptographic purposes on the internet, TLS 1.2 by itself is more than good enough. (if someone could demonstrate that it wasn't, then it would be considered broken and it would be retired). Your question is sortof similar to asking "Will I cross the street faster if I push the crosswalk button three times instead of once?"

Moreover, I suspect that if I can break your TLS (for example by hacking into your server and stealing the server's private key) then I can probably do the same for whatever other protocols you're layering.

Unless you have a specific reason why you believe TLS to be insecure in your environment, or you have some exceptionally high national military type security needs (and have hired a team of crack cryptographers to engineer something better than TLS, the state of the art in cryptography), then you won't get any practical benefit out of layering. Even then, you'd might as well just replace TLS with a single stronger protocol rather than layering.


Where to check if a specific OS / service is vulnerable despite being patched, because of a vendor's decision? I need to discuss how to mitigate the risk of services which are vulnerable despite being patched (typically because they are not maintained by the vendor anymore, or because the vendor does not want to issue a patch, or for other reasons beyond control).

I know how to check if a machine is vulnerable (via a patch inventory system, a vulnerability scanner, ...)
I know where to look for details regarding vulnerabilities (OVAL, CVE, NVD, ...)

What I don't know is where to check if a given OS, service or application will be vulnerable despite being patched and hardened, because of a vendor's decision.

An example would be Windows Server 2003 which is not maintained by Microsoft anymore and may be vulnerable to Windows-wide vulnerabilities discovered after the EOS. Having this knowledge, it is easier to prioritize the contingency in order to mitigate the risk.

Is there such a resource available?

This is exactly what vulnerability scanners do. They just aren't great at it (none I've ever seen detect ALL unsupported software, and I doubt they will any time soon!).

Generally all vendors have their own way of disseminating this information - if they do at all - which is a major pain for asset owners and security specialists. For vulnerabilities themselves I find CVE's to be a great resource (through the NVD website for example) but it does not cover support. I'm not aware of a central repository for this information.

the CVE Details  website is great resource for searching specific version numbers for known vulnerabilities. However in this context "unsupported" is not one of them.

One example I can think of here about "vendor decision" is CVE-2015-0008/MS15-011 which remains unpatched on Server 2003 even though it was supported at the time.


Wireless Access Point security with TEW-638APB We have a large network that is private, with both eithernet and Wifi access.  We'd like to make another smaller network that is wireless, open, and possibly throttled, and not able to access our larger network with file and print servers.  Obviously, a good way to do this would to be to add a wireless router into the mix with DHCP enabled.  However we have an extra TEW-638APB and really appreciate the wireless range on these.  I'm wondering if it would be possible to set it up to do the job?

This is a pretty common configuration.

TRENDnet's kb has this to say (not copied here for copyright reasons).  Basically, yes it is possible.  The AP can support up to 4 SSIDs that are on different VLANs.  Just set your routing to have your "guest" VLAN only route to the internet, and not to any of your internal segments.


Secure communication between C# client and Java server, using certificates I want to secure communication between simple C# WPF client and Java server. Now i have already implemented raw sockets communication, and my security is based on:

server send RSA public key to client ->

<- client send back encrypted with RSA public key, AES key for future communication.

but i'm not assured that someone (the men in the middle) will catch and replace my public key. I suppose that for avoiding men in the middle i need to use CA certificates. Can someone help me, how to make a SSL connection between Java server ans C# client using certificates ? And where i can find some certificates for testing purpose.

I suggest that you use a SSL/TLS library for C# (SslStream? OpenSSL? ask on Stack Overflow if you want more recommendations) or Java (SslSocket? BouncyCastle? ask on Stack Overflow if you want more recommendations).

You can use TLS to encrypt the entire communication.  As @ordag says, don't try to invent something on your own.  This is tricky stuff.  TLS has been carefully vetted, and there are many existing implementations, so it will be pretty easy to use.

As @symcbean says, you can easily create your own self-signed certificates (e.g., with OpenSSL), which should fine for your purposes.

Read the following questions, too: What's the difference between SSL, TLS, and HTTPS?, Best tool for testing an SSL connection in my application.


If the IP address's GPS coordinates is far enough from me, could a hacker be on my line? My current IP address is linked to a GPS location 40 km from my location. I found the same GPS coordinates using both iptrackeronline and maxmind. If this means that there is a hacker on my line, this may mean that he sees this too. What am I to do if I have a hacker on my line?

Getting the location from the IP address is simply done by a database lookup and different providers might use the same database or at least similar sources. This explains why you find the same physical location with different providers.

The accuracy and precision of the entries within the database varies widely. To cite from whatsmyipaddress.com: How accurate is IP GeoLocation?:

  Accuracy rates on naming the city from an IP address vary between 50%-80%.
  Determining the nation of an Internet user is 95%-99% percent accurate, for reasons that have to do with how IP addresses are allocated and registered.
  Determining the physical location of the user, down to a city or ZIP code, is trickier and less accurate.


Calculating password entropy? Whenever I look at password entropy, the only equation I ever see is E = log2(RL) = log2(R) * L, where E is password entropy, R is the range of available characters, and L is the password length.

I was wondering if there are any alternate equations for calculating entropy, which factor weak passwords into the equation. For instance, passwords with sequential characters (0123456789), common phrases (logmein), repeating words (happyhappy) or words with numbers appended (password1) would all receive a lower entropy grade due to their various shortcomings.

Does such an equation exist? If so, is it commonly used in the security field, or do people tend to stick with the "standard equation"?

There are equations for when the password is chosen randomly and uniformly from a given set; namely, if the set has size N then the entropy is N (to express it in bits, take the base-2 logarithm of N).

For instance, if the password is a sequence of exactly 8 lowercase letters, such that all sequences of 8 lowercase characters could have been chosen and no sequence was to be chosen with higher probability than any other, then entropy is N = 268 = 208827064576, i.e. about 37.6 bits (because this value is close to 237.6).

Such a nice formula works only as long as uniform randomness occurs, and, let's face it, uniform randomness cannot occur in the average human brain. For human-chosen passwords, we can only do estimates based on surveys (have a look at that for some pointers).

What must be remembered is that entropy qualifies the password generation process, not the password itself. By definition, "password meter" applications and Web sites do not see the process, only the result, and uniformly return poor results (e.g. they will tell you that "BillClinton" is a good password). When the process is an in-brain one, anything goes.

(I generate my passwords with a computer, not with my head, and I encourage people to do the same.)


Stealing SSL Session Id can cause any harm? In SSL handshake, server shares the SessionId in Server Hello message and since there is no encryption yet, so anyone can capture this ?

Is it risky ?

This is not risky.

The session is in ServerHello references a previously established TLS session which can be chosen to resume by the client.

This makes it possible to leave out otherwise repeated handshakes, but with just the session id, an attacker cannot do much.

Even if the attacker previously witnessed the handshake that is now skipped, they would know the configuration parameters (as they are the same) of the session - but not the ephemeral keys that got exchanged during the last handshake, as their transmission was secure in the first place (assuming that TLS was used in the right version and with the right configuration).

The misunderstanding might come from the multiple use of „session“: in TLS, there are sessions - and in HTTP(S), there are sessions as well, sometimes. The session id in the ServerHello is not an HTTP(S) session cookie, though. They are different things altogether.

A TLS session has nothing to do with the session you may know from “http session hijacking”.


What is the name for this type of URL attack? Say a PHP page accepts URL path as a POST parameter (like the answer to this question):

$path = $_POST['url_path'];

file_get_contents('http://example.com' . $path);

A maliicious user POSTs url_path as @evil.com/stuff.html.

As the code takes the user input and appends it without a delimiter to end the username section (e.g. ?, /), $path ends up as:

http://example.com@evil.com/stuff.html

which means username example.com accessing http://evil.com/stuff.html.

Is there already a specific name for this type of attack? The closest I found was Semantic URL attack, however this seems more syntaxic. It appears to be a type of URL obfuscation, however as it is on the server and not visible to the end-user in the address bar it is a slightly different type of attack to Phishing.

Note: The code is also vulnerable to other types of domain manipulation such as appending something to change the interpretation to another top level domain (e.g. POSTing .uk to change example.co). So on that basis I would just call it a general string concatenation flaw rather than an attack specifically relating to URLs - however, I'd like to put this out to the floor.

Server Side Request Forgery (SSRF) where the server makes a web request to a user controlled URL. MITRE has assigned this as CWE-918.


Taking password letters not whole one, is this secure? I found a site which is taking password letters from their users not the whole password.

Is this secure? Or do they have saved password as salted hash (MD5)?

This is just really terrible. The only way this can work is if they encrypt your password instead of hashing it. Or even worse just store it as plain text. I would never sign up for that service ever. <update>As @lynks stated the above is probably not true when using HMS. However if only a username is asked before this step. And the password is a fixed password of the user the rest of my answer should still stand.</update>

Also the fact that you only have to guess 4 characters (instead an entire password) is just stupid. Also those blocks may indicate the length of the password which is also really crap (if that indeed is the case).

Not only that, but they also show you on what place the characters should be added which is even more stupid, because you can now make better guessing of the characters being used based on the distribution of characters in words / sentences.

All in all what you have there is mega crap and should not be used... ever. I would go as far as saying that the person who tought this would be a good idea should stop doing stuff like this and start looking for another job.

In fact it is so bad it wouldn't surprise me if the actual characters that are hidden by those blue blocks are just hidden and stored in the source of the page ;-)

As a side note based on the last paragraph of your question:

  or do they have saved password in salted hash (MD5)?

MD5 should never be used for hashing passwords. It is waaaaay too fast for that purpose.


Linux forensics questions (ssh config, user activity) I'm a linux noob and have an assignment I'm a bit stumped on... We're given VirtualBox (vdi) images and raw files for the state of a linux server at 3 different dates.

We're supposed to answer:

what order did people hack in and give themselves accounts?
did a certain user mess up ssh functionality for everyone?
what other shenanigans have people been up to?
tips: check out /var/log/ and the programs extcarve and Bulk Extractor

And we're not taught how to do any of this :(

My thoughts / questions:

I'm looking at the /etc/shadow file now for user accounts. This should work right? I'm not missing anything if I do it this way?
Googling yielded the existence of a /etc/ssh/ssh_config file, but I have no idea what to look for in here... Any tips? Is it possible to tell who modified a file? Or do I need to go through every user's /.bash_history to check? x_x
I looked at the /var/logs directory and didn't see anything immediately useful. Any ideas what he may want us to look for in here?
I'm not sure what other shenanigans he wants us to look for. extcarve and Bulk extractor sound useful if you're trying to put together statistic on the entire drive, or maybe a file was deleted and you want to check on what's left of it? Do you think that's what he has in mind?

Sorry for the noob questions, but thanks for any help!

Welcome to the Wild and Wonderful World of Forensics! So you have your images. Before you do anything else make sure you get a checksum of the disk images and also make a copy of them. The copy means you can work on it instead of the original, allowing you to roll back to the original state if you accidentally change anything on the system. The hash will allow you to validate your working copy against the original. For this purpose, just about any hashing algorithm should be sufficient.

In order to really understand what's going on, or went on, in a system you must first figure out what it was. It is obvious that you're looking at some form of UNIX derivative, probably a Linux, but you should attempt to determine which family it belongs to. Is it a RedHat, a Debian, a BSD, Solaris, or even something completely off kilter like AIX or HP-UX?

The type of system you're inspecting will determine exactly what log files you need to browse and how you determine where configs are stored. The advice to start in /var/log is pretty good, that is the default log location for Linux systems. Depending on the specific flavor you may also find useful information in /var/log/audit from the auditing subsystems (if installed) or the RBAC/MAC systems like SELinux or AppArmor.

Answering your specific config questions, account information is stored in a number of files: /etc/passwd, /etc/shadow, /etc/group, /etc/gshadow. The construction of each is well understood and easily researched. For other services the specific config files, and their construction, will require both research as well as experience based understanding on your part.

Performing a proper post-incident investigation requires that one have a deep understanding of the system you are analyzing. You will not only have to understand what files to look at, but also what files imply what, how different configurations can interact with each other, and (when carving) how a filesystem is actually constructed. Having been thrown into the deep end, as it were, will be tricksome, but can be gratifying. It's not without merit that the mantra of the Offensive Security team is Try Harder™. While this isn't exactly offensive work the principles definitely apply here.


What is the recommended procedure for migrating 2-factor authentication to a new device? A common scenario encountered with 2-factor authentication on mobile devices using, for example Google Authenticator, is migrating authentication to a new device. A natural way to accomplish this is to disable 2fa for all accounts, and then set it up again for the new device. This ensures that the old device cannot be used to generate keys, that the new device can be configured for 2fa (which would not otherwise be possible), and avoids having to store or locate additional recovery keys.

Is this procedure — disabling and re-enabling — the recommended approach? Other than the short period during which 2fa is disabled, are there other risks or drawbacks to this approach?

What is the recommended procedure for migrating 2-factor authentication to a new device?

I don't think that disabling 2FA entirely is a recommendable approach, but this update process is specific to the site/service implementing 2FA.

In the case of Google 2FA, I don't think disabling it is required, or even recommendable, if only for the practical implications.[1]

To migrate Google Two Factor Authentication to a new device, I'd recommend the following:

Make sure to have generated and saved/printed a set of emergency codes, in case anything goes wrong. You can also register a phone number to send codes to.[2]
Configure a new Authenticator app via your account preferences, using your old device to log in, and configuring Authenticator on the new one.[3]

If you are reinitializing the device that has Authenticator installed, you could do one of the following:

Login before wiping the old device, re-install and update the Authenticator prefs in your account, or
Use one of the aforementionned emergency codes to log in and update your prefs once the device is re-installed.

1: If you have set up app-specific passwords (e-mail client on your phone or computer, for example), they won't be able to connect while 2FA is turned off, and you may need to reconfigure all of them once you switch it back on.
  2: This will also prevent any catastrophic scenarios where a phone is lost along with its backups
  3: As far as I am aware of, Google 2FA does not allow you to have two Authenticator app's configured for your account. Once the new one is succesfully set up, the old one won't work.


Are these log messages in my nginx server evidence of an attempted attack? My AWS Elastic Beanstalk server is often reporting Environment health has transitioned from Ok to Severe. 100.0 % of the requests are erroring with HTTP 4xx. This happens almost daily for a few minutes at a time. I also found the logs below in my ngnix error logs. What are these? Are they some kind of DOS attack? I looked up the ip addresses of a few of them and found that they're coming from an Amazon data center. If this is something I'd rather prevent, is there a way to do so without blocking traffic to legitimate users?

The server health and the timestamps on these errors do not seem to correspond.

> 2019/07/31 16:53:51 [error] 26340#0: *242697 client intended to send
> too large body: 4294967295 bytes, client: 172.31.12.183, server: ,
> request: "GET /msdn.cpp HTTP/1.1", host: "35.183.225.3" 2019/08/01
> 01:45:12 [error] 26340#0: *250681 client intended to send too large
> body: 4294967295 bytes, client: 172.31.30.80, server: , request: "GET
> /msdn.cpp HTTP/1.1", host: "35.182.179.84" 2019/08/08 13:18:17 [error]
> 26340#0: *414178 client intended to send too large body: 4294967295
> bytes, client: 172.31.12.183, server: , request: "GET /msdn.cpp
> HTTP/1.1", host: "35.183.225.3" 2019/08/08 16:12:57 [error] 26340#0:
> *416902 client intended to send too large body: 4294967295 bytes, client: 172.31.30.80, server: , request: "GET /msdn.cpp HTTP/1.1",
> host: "35.182.179.84" 2019/08/19 20:38:21 [error] 26340#0: *662525
> client intended to send too large body: 4294967295 bytes, client:
> 172.31.12.183, server: , request: "GET /msdn.cpp HTTP/1.1", host: "35.183.225.3" 2019/08/20 04:01:45 [error] 26340#0: *669529 client
> intended to send too large body: 4294967295 bytes, client:
> 172.31.30.80, server: , request: "GET /msdn.cpp HTTP/1.1", host: "35.182.179.84" 2019/09/17 08:16:14 [error] 26340#0: *1282762 client
> intended to send too large body: 4294967295 bytes, client:
> 172.31.12.183, server: , request: "GET /msdn.cpp HTTP/1.1", host: "35.183.225.3" 2019/09/17 11:22:36 [error] 26340#0: *1285562 client
> intended to send too large body: 4294967295 bytes, client:
> 172.31.30.80, server: , request: "GET /msdn.cpp HTTP/1.1", host: "35.182.179.84" 2019/09/24 16:37:53 [error] 26340#0: *1442933 client
> intended to send too large body: 4294967295 bytes, client:
> 172.31.12.183, server: , request: "GET /msdn.cpp HTTP/1.1", host: "35.183.225.3" 2019/09/24 23:22:03 [error] 26340#0: *1449079 client
> intended to send too large body: 4294967295 bytes, client:
> 172.31.30.80, server: , request: "GET /msdn.cpp HTTP/1.1", host: "35.182.179.84"

AFAICS don't worry.

This seems to be an Attack which does not affect NginX.  Also I do not think this is a targeted attack, as I can see similar requests on my webservers, too.  Hence it looks like some of those automated vulnerability scans to exploit something somewhere.

GET requests do not have a body, so a content-length of -1 is for representation selection.  So this were just some small GET requests and not the indication, that somebody tried to upload 4 GB to your site.

For GET this type of request does not mean "I want to send you a 4294967295 bytes body", instead it is the request for "please answer me with 4294967295 bytes", which your server apparently declined.

Also 4294967295 is 0xffffffff or -1 for 32-bit signed integer, so on 32 bit it might trigger some DoS or any sort of NULL pointer trouble in the attacked script.  Even on 64 bit it might be that msdn.cpp mis-behaves when it interprets the content-length.

Please note that with a quick glimpse I did not find any source for some possible file named msdn.cpp on the web.  So it is unlikely that it is widely deployed.

Hence such requests might be a way to try to locate some already installed backdoor which obfuscates as msdn.cpp.  If it resides on some NAS/IoT device which is behind a dialup line, it might change the IP address daily, so perhaps somebody wants to search for it by scanning the whole web.

There are only roughly 3 billion IPv4 addresses to scan.  So just 50 machines with a query rate of 1000 requests per second can scan the entire IPv4 space in just one day, and 1000 req/s (of 1 KB each) only need a moderate uplink of around 1 MiB/s per machine.

There is no need for a BotNet to do this.  10 VMs cost you around $1 per day.  Easy.


Is it secure to use MD5 to verify the integrity of small files (less than 15kb)? I know that collision for MD5 has been documented since the 90s and that digital certificates based off of MD5 has been demonstrated to be completely compromised back in 2010 but how effective is MD5 in ensuring that small amounts of data have not been tampered with?

I have some small text files that are a few pages in size (let's say 15kb in size). I've been using SHA-256 on them but it would be much more convenient to be able to use MD5 instead.

How secure would MD5 be as a hash digest for these small 15kb text files? Would a malicious party be able to produce collisions for such a small amount of data or does the small size make this a difficult endeavor?

The size of the input is irrelevant. In fact, because of the birthday paradox, collisions are guaranteed as soon as the size of the message exceeds that of the hash. The best way to avoid collisions is to use a stronger hash which is not vulnerable to them, such as SHA-2. However, you are describing a more difficult attack than a collision attack, called a preimage attack, which MD5 is safe from.
There are three types of attacks* that result in having two files with the same digest:

1st preimage - Find an input that resolves to a specific hash.

2nd preimage - Modify an input without changing the resulting hash.

Collision - Find any two distinct inputs that have the same hash.

These are vulnerabilities when they can be carried out more efficiently than by brute force search. Collisions can still occur naturally, and in fact they are guaranteed with any non-trivial amount of input due to the pigeonhole principle, but hashes are designed to make it difficult to intentionally perform. For a hash with an output the size of MD5's, the chance of a random, accidental collision is extremely low. Even if you hash 6 billion random files per second, it would take 100 years before you get a 50% chance of two hashes colliding. MD5 is great for detecting accidental corruption.
A strong n-bit hash function is designed to have a security level of 2n against both 1st and 2nd preimage attacks, and a security level of 2n/2 against collision attacks. For a 128-bit hash like MD5, this means it was designed to have a security level of 2128 against preimages and 264 against collisions. As attacks improve, the actual security level it can provide is slowly chipped away.
MD5 is vulnerable to a collision attack requiring the equivalent of only 218 hash invocations instead of the intended 264 to exploit. Unless the attacker generates both files, it is not a collision attack.† An attacker who has a file and wants to maliciously modify it without the hash changing would need to mount a 2nd preimage attack, which is completely infeasible against MD5 with modern technology (the best attack has a complexity of 2123.4, compared to MD5's theoretical maximum of 2128). Collision attacks are relevant in different situations. For example, if you are given an executable made by an attacker without a backdoor, you may hash it and save the hash. That executable could then later be replaced with a backdoored version, yet the hash would be the same as the benign one! This is also a problem for certificates where someone could submit a certificate for a domain they do own, but the certificate would intentionally collide with one for a domain they do not own.
It is safe to use MD5 to verify files as long as the stored hash is not subject to tampering and can be trusted to be correct, and as long as the files being verified were not created (or influenced!) by an attacker. It may still be a good idea to use a stronger hash however, simply to prevent a potential practical preimage attack against MD5 in the future from putting your data at risk. If you want a modern hash that is very fast but still cryptographically secure, you may want to look at BLAKE2.
* While there are other attacks against MD5 such as length extension attacks that affect all Merkle–Damgård hashes as mentioned by @LieRyan, these are not relevant for verifying the integrity of a file against a known-correct hash.
† A variant of the collision attack called a chosen-prefix collision attack is able to take two arbitrary messages (prefixes) and find two values that, when appended to each message, results in a colliding digest. This attack is more difficult to pull off than a classic collision attack. Like the length extension attack, this only applies to Merkle–Damgård hashes.


How to encrypt data and login with same password - revisited I read previous posts about the topic of how to encrypt data and login with the same password/key, however the concept still seems unclear to me, especially which data it is safe to store in the database.
This great Post advised the following 2 possibilities:

1: You could derive the encryption key from the password with PBKDF2, and then use a normal cryptographic hash (e.g. SHA-256) of the key as the password hash for logging in.
2: Or, if you want to be extra careful, derive both the encryption key and the password hash from an intermediate key derived from the password using PBKDF2.

I created a diagram of my understanding for the second possibility. Do you think that I have gotten it correctly? Can you maybe think of a better way of implementing it?
N.b.

I replaced PBKDF2 with scrypt here, but the concept is the same.
Salts are 16 random bytes generated with urandom

UPDATE
I discovered that the PyCryptodomex (Python) library offers a scrypt function in which I can specify how many hashes/keys I want from a single input key. For example, if I want 2 256bit long output hashes, I can specify that and scrypt will create a 2 * 256bit = 512bit key and simply divide it by 2, resulting in 2 256bit long hashes. I can then use one hash for encrypting the data and one for authentication, thus eliminating the need for the second level of scryt and SHA256 hashing in my previous concept. I created a visual description of the new concept and I'd appreciate your opinion about it :)

Summary
Depending on the library and KDF you use, I guess that either concept 1 or concept 2 (which is essentially the same, but more compact) will work.

If by "PW - Hash + Salt", you mean the derived key, your understanding is correct.

I've seen a few variations

Use of different salts each time you use one (i.e., Salt 1 and Salt 2 in your case). There is some theoretical benefit with this, but I don't consider it big enough.
Use the PW as input in the second round (of hashing / key derivation) too. Again, I'm not sure it is a big enough benefit, but the cost is negligible.
Use of a large number of rounds with the key derivation function (you probably implied this). This is important since the whole idea is to increase the work factor involved.
OWASP recommends 32 or 64 byte salt here: https://www.owasp.org/index.php?title=Password_Storage_Cheat_Sheet. IMHO benefits at this time are theoretical since iterating over even 16 bytes is not a trivial job for an attacker.

In general, I don't like the logic that says "let's do it just in case" (i.e., we let our fear decide without assessing the risks vs benefits). So I'm skeptical about the whole idea of the Intermediate Key. YMMV. :)


Does Hole Punching put added Vulnerability on the Client Side Websites and servers frequently get hacked when a vulnerability is present in the server side code. Does UDP or TCP hole punching put that risk onto the users of the application when the connections become peer to peer?

Take for example the clearly vulnerable practice of execing an unsanitized input. Obviously this practice would not be done by design, but this kind of thing happens far too often by mistake. In a traditional setup, if user_a interacts with user_b we would first have user_a send input to the server, the server would process the data, and then send something to user_b. With hole punching, we get a peer to peer between the two users and so user_a sends data to user_b, and user_b processes it for himself. If a vulnerability was present in how the input was processed by the server (in this case execing it without sanitizing), the vulnerability could just as easily be present on the client side, allowing user_a to inject and directly run code on user_b's machine.

If a severe vulnerability were to be found in a hole punching application (Skype and other VoIP services, some multiplayer realtime games, et al), is there anything stopping a malicious hacker from having a direct backdoor into the system of any user they interact with? If not, are peer to peer programs unnecessarily risky? Is there a catch-all way to mitigate this risk for people creating these application? And what about people who use these applications? Do programs like Skype leave users at an undue level of risk (more so than typical)?

The question depends heavily on the software security architecture used but yes there have been cases where the peer to peer software implementation did allow bad actors to access files on the hard-drives of other people participating in the peer to peer network. Likewise some of these designs also allow participants, or ISP's, to identify the files other users are sharing remotely.

This said it doesn't mean these applications are necessarily unsafe some could be very safe but any time you add an application which listens to the Internet (or other systems) you are increasing the attack surface of the computer. This risk is not unique to peer to peer software but peer to peer software is more likely to have listening ports.

Again it really depends on the security models used more than anything.

In general all of the security issues with securing a non peer to peer protocols will apply to peer to peer protocols as well. This just happens to be a many to many set of connections rather than one to one or one to many.

Again, they could be really secure but every implementation will be very different.


smime - is encrypting reply to a signed mail without own cerificate possible? From what I understood, was that having a S/MIME certificate allow me to sign emails, and any recipient of my signed email would be able to encrypt emails send to me.

Is that correct? or does the recipient of my signed email also needs their own S/MIME certificate in order to encrypt emails to me?

Thanks in advance

While it is technically correct that you can encrypt content for a recipient key that is contained in an S/MIME certificate without having a certificate of your own, this is not the same thing as sending an S/MIME email.

While the RFC allows for Enveloped only S/MIMES it does state:

  It is important to note that sending enveloped
     but not signed messages does not provide for data integrity.  It is
     possible to replace ciphertext in such a way that the processed
     message will still be valid, but the meaning can be altered.

In production mail clients, S/MIME messages are hence always signed - whether or not they are encrypted. While you do not need a key pair and certificate for the encryption part of S/MIME, you do need one for signing that encrypted message.

  In my opinion both parties needs S/MIME certificates in order to communicate securely. Is that correct?

Yes, and for good reason: S/MIME is not only about confidentiality but also about authenticity; when an E-Mail is not signed properly, authenticity is out of the window.


What is the best method for key exchange? I have application working between server and client. All messages must be encrypted and decrypted. I use AES algorithm and I want method to hide the key.

I found Diffie-Hellman algorithm, but my problem is that I didn't find its source code. If the is another method for key exchange, please tell me.

This is exactly what SSL/TLS are supposed to do. SSL/TLS uses asymmetric encryption just in the handshake to exchange the symmetric key that is used from that moment on to encrypt the communication. The key exchanged is called a masterkey and is only used for that session. The lifetime of a session can be configured.

The process is basically this when you use HTTPS, for example:

Client establishes connection (three-way-handshake -> TCP handshake, after this the SSL/TLS handshake starts)
Server sends certificate (which includes the public key)
Client sends masterkey encrypted with the servers public key
Masterkey is used for that session

This is of course just a broad description of the protocol and only applies when the client itself has no certificate which is the case in most scenarios.


NTLMv2 resistance to bruteforcing I have a question regarding NTLMv2 resistance to password bruteforcing.

I know that some modern graphic processors (like Radeon 6990) are able to calculate billions hashes per second and crack NTLM hashes within minutes and hours.

I would like to evaluate how NTLMv2 hashes are more resistant to password bruteforcing comparing to NTLM.

Given that roughly:

NTLM hash = MD4(password)

and

NTLMv2 hash = MD5(MD5(MD4(password) + X) + Y)

Is it correct to say that attacking NTLMv2 hashes via password bruteforcing is just 3 times longer than attacking NTLM hashes?

Actually it is closer to 4 or 5 times, because MD5 is somewhat more expensive than MD4. It won't make it strong, though; the "benefit" of switching from NTLM to NTLMv2 is canceled by waiting one year and then buying a bigger GPU.

Windows systems stick to NTLM variants because of backward compatibility, but they are rather poor password hashing methods. See this answer on how passwords ought to be hashed.


What is the practical approach for CIS Benchmark (Ensure permissions on all logfiles are configured)? I am trying to harden RHEL with CIS benchmark. One of the items states the following:

Ensure permissions on all logfiles are configured
Description: Log files stored in /var/log/ contain logged information
from many services on the system, or on log hosts others as well.
Rationale: It is important to ensure that log files have the correct
permissions to ensure that sensitive data is archived and protected.
Other/world should not have the ability to view this information.
Group should not have the ability to modify this information.

My questions:

What kind of system logs are considered as sensitive data?
Is it acceptable if there are deviations because some system logs permission will be reset during boot?

For the first, it depends on data. One must know its OS and applications. Some apps leak users' passwords when running in debug for example.
For the latter, if you are investigating and you don't have logs you don't have any clue what was going on. You can always send logs to a remote host if you don't want to save logs locally, sending logs to a remote node is always good choice. The next step what an attacker would do is to break into a log host and delete the traces.


Why does Kaspersky warn me about PHP trojans on a Mac? I have a question regarding PHP Trojans on a Mac. I have downloaded a website's home directory which has been hacked to my local Mac. On this Mac I am running Kaspersky Internet Security which ran a check on the downloaded folder and it indicated that there were 3 PHP trojans (Trojan.PHP.Agent.pg) in this folder.

My understanding is, that the PHP Trojan can only be harmful to a system, if it can be executed. Therefore:

Can my MacBook interpret the PHP code and actually get harmed?
If not, is Kaspersky just warning me due to pattern matching? Would it therefore also warn me for Windows Maleware - or is it just one common database and they do not differentiate which operating system one is using?

PHP being a scripted language needs to be interpreted and run by the script interpreter.

Its like downloading a malicious VBScript or bash script, until you execute it, by passing the script to the interpreter, it remains benign.

Can you provide more details as to what the AV detects? It usually gives a value like Trojan.WXYZ which then links to a website where more information is included.

Caveats: This is assuming there isn't another exploit at work, for example the windows exploit where a malicious .lnk file would be executed simply by viewing it in explorer.


What is the exact difference between SAMLp and WS-Trust? They seem to be similar on the surface, but I'm not sure how deep the differences are.

Can someone explain to me the difference between SAMLp and WS-Trust?  ADFSv2 gives me a choice between these options and I'm not sure which to choose.

WSTrust is a standard which is specifies how to encrypt/decrypt/sign SOAP messages and how to bind security tokens to a SOAP message.

SAML tokens are an example of a security token that could be bound to a SOAP message.

The SAML protocol provides a standard way to acquire, release and renew SAML tokens using SSL to protect communications and the XmlDsig standard to protect the token.


How to prove to friend I have solution to puzzle (without revealing it)? My friend Paul set me this arithmetic puzzle:

  To make 21 from the numbers 1, 5, 6, 7. You can use the operations add, subtract, multiply, divide as well as brackets. You must use each number once.

After I solved it myself, I set the puzzle to another friend Ollie. He thinks it's impossible. How can I prove otherwise to Ollie—without revealing the solution? Can it be done without a trusted third party?

Also, how should I show my solution to Paul? I suspect he might not have one himself. If that's the case, I'd rather not tell mine—or at least expose him in the process.

You can use a zero-knowledge proof to demonstrate that you have a solution to a general class of problem. (For example: "I can efficiently find the prime factorization of any number. I won't tell you how I do it, but I'll prove my ability by quickly factorizing any number you throw at me.") However, here, the solution is how you solved the problem; it's tautological to say you can't show how you solved the problem without giving away how you solved the problem.

You could secretly but provably declare your solution via cryptographic commitment. For example, you publish a hash of your solution and then later publish your solution. That allows you to prove that you had the solution for at least as long as the hash has been public. However, it only becomes clear that the secret you committed is a valid solution after you make the secret public. (This approach will allow Paul to compare his (non-)answer with yours -- simply compare hashes, using a standard format -- but it will not prove anything to Ollie.)

For Ollie, you need some way of demonstrating a solution to a different problem that maps onto the actual problem in some provable way. I briefly considered some kind of fully homomorphic encryption, but I don't see how you could homomorphically demonstrate the problem in a way that 1) doesn't allow your friend to decrypt it and 2) still allows your friend to verify the result. For example, consider an attempt at an iterative zero-knowledge proof with a homomorphic scheme that can encrypt (Enc) with a public key and decrypt (Dec) with a private key:

Show your friend Enc(1), Enc(5), Enc(6), and Enc(7) in a random order, and separately show Enc(21). At this point, your friend may stop these steps, demand the private key and verify that you are not lying about the values of the numbers. (This prevents you from using the wrong values, since you'd probably get caught if you played enough times.)
Perform the necessary mathematical operations on the first four encrypted values to produce 21. (This step gives away the necessary operations, which is suboptimal, but it does not reveal which values go with which operations.) At this point we can no longer give your friend the decryption key, or else he will be able to view the entire solution.
Perform a homomorphic equality test of the result from step 2 against the Enc(21) from step 1. This produces an encrypted boolean.

There is no way to allow your friend to decrypt the boolean without also allowing him to decrypt every other value, which would reveal your solution entirely. Therefore, there is no way to verify to your friend the correctness of your solution.


Securing API keys for a Twitter account for a program to be run on other PC's I've created a method to allow my project EditVideoBot to be 'decentralised', where rather than the program processing and uploading all video editing requests on my own central server, users can volunteer to run this program on their computer and take some of the load off my server.
I'd like to be able to start distributing the program, but I have one major concern:
In order for the program to upload the final edited video to Twitter, it needs my Twitter API keys. Obviously, I need a way to encrypt the keys so they're not accessible to the user running the program (otherwise they have full access to my account).
The bot is programmed in Python. I don't really know where to begin with keeping the keys secure if the program is running on another computer.
Edit: If there was a method with the Twitter API to generate keys that have restricted priveliges (e.g. they can only access the media-upload endpoint but not any other endpoints) that would be the best method. (like with Cloudflare's API). Then, they'd forward the returned media-id to my server where it has full access and posts the tweet attaching the media ID. I don't think they have that capability though, so another method is needed.

As specified, the problem is completely impossible. You can not, and should not attempt to, make a program - much less a script - do something its user can't see. There are many ways the attacker could break this. They could just read and analyze your python scripts (even if compiled, .pyc is easy to decompile). They could debug the program as it runs. They could intercept the network requests it makes to Twitter and pull the key out of there. They could replace the Python HTTPS library with one that wraps the official API and also logs a copy of the requests and responses to plain text. They could load your program as a library of another python program and programmatically examine its contents... Seriously, people keep asking about this and it just can not be done.
As @Limit's comment says, your best bet is to expose an API from your server, and have your script call that with whatever it wants to put on Twitter. You can then validate the message before posting it, rate-limit each client's uploads, prevent abuse of the account such as deleting tweets or using DMs, and revoke third-party access to the account entirely just by disabling the endpoint on your server. This isn't enough to totally prevent abuse, but it'll do a lot better than any method of trying to hide secrets in a Python script.
Another option is to use unique Twitter accounts per user. Either require the users to create/supply their own account (which your server then learns about and monitors for uploads, perhaps with specific tags or mentions), or automatically create one (well, as automatically as Twitter will let you get away with) for each node and don't use them for anything else (the user running the node would still have full access to the account in each case, but not to any other accounts).
Or you could just have the nodes send the resulting videos to your server, and your server decides what to do with them. This is basically the same as the first viable approach, except the new endpoint is not "send video here to upload to Twitter", it's just "send video here when done" and you-the-server-owner decide what to do with it then.


SHA & MD5 Collisions for shorter strings This is not for passwords. I understand that MD5 and SHA-512, etc... are insecure because they can have collisions.
However, is it still possible to have a collision if the string length is less than the hash size (i.e. MD5 is a 32 character hash)? So if the string is less than 32 characters, is it possible to still have collisions?
For example, to authenticate, I was thinking it would check length of password and that it matches say the MD5/SHA password hash.
So take the MD5 hash for the input: somestring.
The MD5 hash is:
1f129c42de5e4f043cbd88ff6360486f
The input string is 10 digits long. So if I wanted to check it was valid, my parameters would be (pseudo code):
if MD5(“somestring”) = 1f129c42de5e4f043cbd88ff6360486f AND string length = 10, then authenticate string. Else it is not authenticated.
However, this only works if no other string that is 10 characters long equals the MD5 hash of:
1f129c42de5e4f043cbd88ff6360486f
Can shorter strings of the same length collide with MD5 or SHA?
If yes, would they still collide if they were only alpha numeric? What could an an example of this be?

Short answer: collisions don't matter for password verification. The length of the input is irrelevant.

I understand that md5 and sha512, etc... are insecure because they can have collisions.

No, this is wrong. MD5 and SHA-1 are insecure because it is possible in practice to find collisions.
SHA-512 and the other SHA2 variants (SHA-256, SHA-384, etc.) have collisions. We know this by applying a very simple mathematical theorem, the pigeonhole principle. A SHA-512 hash is a 64-byte string. There are strictly more strings of 0 through 64 bytes than strings of exactly 64 bytes. Therefore there exist two distinct strings of at most 64 bytes that have the same SHA-512 hash. This is a non-constructive proof. We have absolutely no clue how to actually find two such strings. SHA-512 is secure because cryptographers have tried to find two such strings for years and haven't gotten any closer. This is the standard for a cryptographic algorithm being “secure”.

This is not for passwords.

Indeed. For passwords, collisions are irrelevant. If there are multiple valid passwords, it doesn't matter. What matters is that an attacker cannot find any of the valid passwords. SHA2 is as insecure as MD5 and SHA-1 for password hashing: password hashing must be slow and salted. It's an unfortunate historical accident that the word “password hashing” includes “hash”: the two have very different properties.

However, is it still possible to have a collision if the string length is less that the hash size. Ie. md5 is a 32 character hash. So if the string is less than 32 characters, is it possible to still have collisions?

Maybe. We don't know. We know that there are collisions among strings that are at most the size of the hash, by the pigeonhole principle, as I wrote above. We don't know whether there are collisions among strings that are exactly the size of the hash: it's possible that all of them map to a different image. Mathematically, that would mean that the hash, with an n-bit output, is a permutation on strings of length n bits. This is extremely unlikely since the proportion of functions on n-bit strings that are permutation is extremely small, but we don't know for sure.
I don't know, without doing more math than I care to at this time, whether it's likely that there are collisions amongst strings that are strictly less than the size of the hash. I won't do the math because it doesn't matter.
By the way, MD5 is actually a 16-byte hash. It's 32 characters long if you write those bytes in hexadecimal, but most 32-character strings are not the hexadecimal representation of anything, so counting 32-character strings is irrelevant.

However, this only works if no other string that is 10 characters long equals the md5 hash of: 1f129c42de5e4f043cbd88ff6360486f

Wrong. This only works if it is impossible to find another string that is 10 characters long and has the same hash. If another string exists but nobody can find it, that's good enough.
MD5, SHA-1 and SHA2 all use the Merkle-Damgård construction. They work by processing the input one block at a time. For each input block, they apply a compression function (a different one for each MD or SHA variant) to the current internal state and the input block to produce the next internal state. At the end, they append some padding to the last partial block and apply the compression function one more time (or more depending on the exact length of the input). The techniques for finding collisions for MD5 and SHA-1 attack the compression function: they are based on finding block content B1 and B2 such that two distinct internal states S1 and S2 map to the same resulting internal state S'. The techniques allow very little control over the content of B1 and B2, so they don't allow creating collisions where the input is less than one block long, perhaps minus a few bits at the end which can be brute-forced to look like the valid start of padding. (In fact, as far as I know, it takes two blocks of input to reach a collision.) So we don't know how to find collisions on strings that are shorter than the block size. Both MD5 and SHA-1 have a 64-byte block, so there is no way to find a collision where one of the strings is shorter than about 60 bytes.
Of course, even if the current attack techniques don't allow finding a collision on short input, it would be stupid to use a known broken hash function. Attacks only ever get better.


Is there anyway to generate a SAM file? I was wondering if it is possible to create a windows SAM file. I already tried using chntpw  but it seems like it only works under a Redhat based linux, ie I was able to blank/change a Windows 7 64bit box on Fedora with chntpw 0.99.5 but I couldn't get it done using Backtrack 5, Ubuntu or Kali. I know I can recover the password with tools like ophcrack or just using pwdump or sth like that to get the NTLM hashes and then decypt it using bruteforce or rainbow attacks, but I want to know if it is possible to just replace the hash in the SAM file (with physical access to the box using a live os), in another words I just want to know how chntpw works and how to do what it does manually.
any help is appreciated...

please note that my problem isn't replacing the password I know several ways to do that, as I said in the title I want to know if it is possible to generate a SAM file from NTLM hashes, I want to know how these NTLM hashes are stored in the SAM file. I don't want any tools, if anything I want a tool that does the reverse of what pwdump does , ie take the hashes and store it to the SAM file (with live cd of course).

Yes, the Windows passwords saved inside the SAM file can be changed, provided the operating system is shut down, you have physical access to the machine and the disk is not encrypted.

A tool that never failed me is ntpasswd, which can be executed from a CD or a USB key and it's very easy to use.


Directories enumeration by errors. CVSS3.1 and remediation Let's consider this scenario.
If I browse to https://www.example.com/existentdirectory the web application shows this error "directory listing is denied".
If I browse to https://www.example.com/nonexistentdirectory the web application shows this error "file directory is not found".
To me this would represent a chance for the attacker to enumerate directories and gain information on the target.
I would assign the following CVSS3.1 vector:
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N
But what about the remediation part? Is it really needed?
Would it be counterproductive advise to use a generic error for both responses?

It is difficult to determine a correct CVSS score as the internal tree structure is not exactly what we can call a "secret".
So the confidentiality impact has to be discussed.
Gaining information about the target makes furthers attacks simpler, but it is not an attack.
This difference is sometimes materialized by the concept of "non compliance" which is different than "vulnerability".
The remediation for this specific example is often to capture and log every error (so you get notified when there is an error 500 and other important ones) and to display a generic error message with a code 200 OK to the browser.
This way, you can also get rid of many web scanners


Buffer Overflow doesn't have enough space for exploit after being crashed So I'm trying to write a buffer overflow for a knowingly vulnerable server application, I want to learn how to do this on my own and just want some direction.

I'm watching it in immunity debugger on the server and have control over the ECX, EBP, and importantly the EIP register. Sending my buffer from the attacking machine crashes the program however I don't have enough space afterwards to place my reverse shell, I do have enough space in the first part of my buffer. I only have a few bytes to figure it out and past this point I'm stuck. Is it possible to place an instruction set in an earlier part of my buffer and use those few bytes to move back to it? Where can I read up on how to do this?

Assuming that you're talking about a vanilla EIP overwrite and not something like SEH, you have two options available to you. Neither of these are what I would consider to be "beginner" techniques as they require a bit of Assembly knowledge though there are some great websites that can help you out.

1. Employ an Egg Hunter. An egg hunter involves sticking a small bit of a shell code in a limited buffer space and then prepending a 8 byte egg (ie n00bn00b) before your shell code. The egg hunter will search through memory for your unique egg and then jump to the shell code that follows it.
Example buffer Variable

buffer = "\x41" *100 + "n00bn00b" + shellcode + "\x90" * 50 + egghunter + offset + jmpback

Where above "offset" could be a simple JMP ESP in a DLL and "jmpback" would be an assembly instruction to jump back into the buffer to somewhere in the nopslide.

While it looks a bit confusing the execution flow is as follows

The offset variable is placed where your EIP overwrite is. This variable simply holds the address to a JMP ESP instruction which then points to the jmpback variable
The jmpback variable holds an assembly instruction (ie \xeb\xc4) which jumps back into the nopsled.
The egghunter shellcode runs and begins searching for the n00bn00b string.
When the n00bn00b string is found it jumps to the shellcode which executes.

The Egghunter is a great way to give yourself some more buffer space.

2. Jumping backwards into the buffer : The second option, if the buffer space is available to you, is to issue some assembly instructions to jump further back into the buffer to execute your shell code. This sounds like the easier of the two options, however it's possible to trash the stack if you're not careful to re-align the stack after shell code execution.

If you have a clean overwrite of another register, such as ECX, it might even be possible to just JMP ECX to fire your shell code.

Note that this answer assumes that things like DEP and ASLR are not enabled on the machine you're attempting to exploit.